Amish Kushwaha - August 10, 2026
llmrouter Part 4: Streaming and Tool Calling Across Every Provider
Part of the llmrouter Series
-
Part 1: LLM Routing Is Infrastructure, Not Application Logic
-
Part 2: Providers — Wiring Up OpenAI, Anthropic, Gemini, and Five More
-
Part 5: Cost Tracking, Prompt Caching, and Production Observability
Two features carry most of what a production LLM application actually does: streaming, so a chat interface feels responsive instead of frozen for three seconds, and tool calling, so the model can act on real data instead of just talking about it.
Both have provider-specific quirks. OpenAI’s streaming chunks aren’t shaped like Anthropic’s. Tool call IDs look different across all three. Write directly against provider SDKs and you either lock into one vendor or end up writing a translation layer yourself. llmrouter normalizes both, so the same loop works whether the model behind it is gpt-5.6, a Claude snapshot, or Gemini.
StreamResult: The Iterator Pattern
router.Stream returns a *StreamResult, not a channel:
stream, err := router.Stream(ctx, req)
if err != nil {
return err
}
defer stream.Close()
for stream.Next() {
event := stream.Event()
// handle event
}
if err := stream.Err(); err != nil {
return err
}Why an iterator instead of a channel? A couple of reasons that matter in practice.
Lifetime is explicit. Close() runs every registered cleanup function and drains anything left on the underlying channel so a producer goroutine isn’t left blocked trying to send. Break out of the loop, defer stream.Close() handles teardown, no dangling goroutines.
Errors have one place to live. stream.Err() is checked once, after the loop. No separate error channel to also watch, no sentinel value to special-case.
Event Types
stream.Event() returns an Event:
type Event struct {
Type EventType
Content string
Delta *Delta
Response *Response
Error error
}There are four event types:
EventContentDelta: a chunk of text content. event.Content holds the delta.
EventToolCallDelta: a chunk of a tool call being assembled. event.Delta (a *Delta, the same struct used for non-streaming partial responses, carrying Role, Content, and ToolCalls) holds the partial state. In practice you accumulate these and act once the stream finishes.
EventDone: the stream is complete. event.Response carries the full response, including usage and, once the router has finished computing it, event.Response.Usage.Cost (Part 5 covers cost tracking in full).
EventError: something went wrong mid-stream. StreamResult.Next() already treats this as terminal internally, so by the time your loop exits you read the failure from stream.Err(), not from a separate EventError case in your switch.
A Streaming HTTP Handler
A Server-Sent Events handler that proxies llmrouter’s stream to a browser client:
package main
import (
"encoding/json"
"fmt"
"net/http"
"github.com/bluefunda/llmrouter"
)
type ChatRequest struct {
Model string `json:"model"`
Messages []llmrouter.Message `json:"messages"`
}
func streamingChatHandler(router *llmrouter.Router) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
var chatReq ChatRequest
if err := json.NewDecoder(r.Body).Decode(&chatReq); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
w.Header().Set("Content-Type", "text/event-stream")
w.Header().Set("Cache-Control", "no-cache")
w.Header().Set("Connection", "keep-alive")
w.Header().Set("X-Accel-Buffering", "no") // disable nginx buffering
flusher, ok := w.(http.Flusher)
if !ok {
http.Error(w, "streaming not supported", http.StatusInternalServerError)
return
}
stream, err := router.Stream(r.Context(), &llmrouter.Request{
Model: chatReq.Model,
Messages: chatReq.Messages,
})
if err != nil {
fmt.Fprintf(w, "data: {\"error\": %q}\n\n", err.Error())
flusher.Flush()
return
}
defer stream.Close()
for stream.Next() {
event := stream.Event()
switch event.Type {
case llmrouter.EventContentDelta:
data, _ := json.Marshal(map[string]string{
"type": "delta",
"content": event.Content,
})
fmt.Fprintf(w, "data: %s\n\n", data)
flusher.Flush()
case llmrouter.EventDone:
payload := map[string]any{"type": "done", "provider": event.Response.Provider}
if event.Response.Usage != nil {
payload["cost_usd"] = event.Response.Usage.Cost
}
data, _ := json.Marshal(payload)
fmt.Fprintf(w, "data: %s\n\n", data)
flusher.Flush()
}
}
if err := stream.Err(); err != nil {
data, _ := json.Marshal(map[string]string{"type": "error", "error": err.Error()})
fmt.Fprintf(w, "data: %s\n\n", data)
flusher.Flush()
}
}
}The X-Accel-Buffering: no header is easy to forget and easy to miss in testing, since it only matters once nginx sits in front of your service. Without it, nginx buffers the response and delivers it in batches, quietly defeating the entire point of streaming. Set it on every SSE endpoint you own.
Tool Calling
Tools are defined with JSON Schema and attached to the request. If the model wants to call one, the response comes back with FinishReason == "tool_calls" and populated ToolCalls on the message instead of plain content.
Defining a Tool
weatherTool := llmrouter.Tool{
Type: "function",
Function: llmrouter.Function{
Name: "get_weather",
Description: "Get the current weather for a location",
Parameters: json.RawMessage(`{
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "City name, e.g. 'San Francisco'"
}
},
"required": ["location"]
}`),
},
}Parameters is json.RawMessage, raw JSON Schema, no reflection-based schema generation involved. What you write is what gets sent.
The Tool Call Loop
Choice.Message is a *Message, worth flagging up front since it trips people up the first time, you’ll be dereferencing it when you append to the conversation history:
func runWithTools(ctx context.Context, router *llmrouter.Router, messages []llmrouter.Message) (string, error) {
tools := []llmrouter.Tool{weatherTool}
req := &llmrouter.Request{
Model: "claude-sonnet-5",
Messages: messages,
Tools: tools,
}
for {
resp, err := router.Complete(ctx, req)
if err != nil {
return "", err
}
choice := resp.Choices[0]
if choice.FinishReason != "tool_calls" {
return choice.Message.Content, nil
}
// Append the assistant's tool-call message
req.Messages = append(req.Messages, *choice.Message)
// Execute each requested call and feed the result back
for _, tc := range choice.Message.ToolCalls {
result, err := dispatchToolCall(ctx, tc.Function.Name, tc.Function.Arguments)
if err != nil {
result = fmt.Sprintf("error: %v", err)
}
req.Messages = append(req.Messages, llmrouter.Message{
Role: llmrouter.RoleTool,
Content: result,
ToolCallID: tc.ID,
})
}
// loop again with the updated messages
}
}
func dispatchToolCall(ctx context.Context, name, argsJSON string) (string, error) {
switch name {
case "get_weather":
var args struct {
Location string `json:"location"`
}
if err := json.Unmarshal([]byte(argsJSON), &args); err != nil {
return "", err
}
return getWeather(ctx, args.Location)
default:
return "", fmt.Errorf("unknown tool: %s", name)
}
}The loop, append the assistant’s tool-call message, append each tool result, send it back, is identical regardless of which provider answers. llmrouter normalizes each provider’s tool call representation into the same ToolCall{ID, Type, Function: FuncCall{Name, Arguments}} shape before it reaches your code.
Parallel Tool Calls
When a response includes multiple tool calls at once, there’s no reason to run them one after another if they’re independent:
if choice.FinishReason == "tool_calls" {
req.Messages = append(req.Messages, *choice.Message)
type result struct {
tc llmrouter.ToolCall
out string
err error
}
results := make(chan result, len(choice.Message.ToolCalls))
for _, tc := range choice.Message.ToolCalls {
go func(tc llmrouter.ToolCall) {
out, err := dispatchToolCall(ctx, tc.Function.Name, tc.Function.Arguments)
results <- result{tc: tc, out: out, err: err}
}(tc)
}
for range choice.Message.ToolCalls {
r := <-results
content := r.out
if r.err != nil {
content = fmt.Sprintf("error: %v", r.err)
}
req.Messages = append(req.Messages, llmrouter.Message{
Role: llmrouter.RoleTool,
Content: content,
ToolCallID: r.tc.ID,
})
}
}If the model asks for a weather lookup and a search at the same time, running them sequentially is latency you didn’t need to spend.
What Differs Across Providers
Tool definitions and the call/result loop work the same way regardless of provider. A few things don’t:
Tool choice forcing. OpenAI and Anthropic both support ToolChoice for auto, none, or forcing one specific function by name. Gemini’s converter doesn’t implement ToolChoice translation at all right now, tool definitions work fine on Gemini, but if your flow depends on forcing a specific function call, route that request to OpenAI or Anthropic instead.
Tool call IDs. Each provider generates its own ID format for a tool call. llmrouter surfaces whatever the provider returns as-is in ToolCall.ID, and you send it back unchanged in ToolCallID on the tool result message. Don’t try to construct or reformat these yourself.
Prompt caching on tool schemas. Anthropic’s CacheControl field can be set on message content to cache large, repeated context, including tool schemas passed as part of the conversation. OpenAI and Gemini handle caching automatically without any explicit annotation. Part 5 covers this in depth.
Multimodal: ContentParts
For messages that carry more than text, use ContentParts instead of Content:
imageMessage := llmrouter.Message{
Role: llmrouter.RoleUser,
ContentParts: []llmrouter.ContentPart{
{Type: "text", Text: "What's in this image?"},
{Type: "image_url", ImageURL: &llmrouter.ImageURL{URL: "https://example.com/diagram.png"}},
},
}Or base64-encoded:
imageMessage := llmrouter.Message{
Role: llmrouter.RoleUser,
ContentParts: []llmrouter.ContentPart{
{Type: "text", Text: "Describe this screenshot"},
{
Type: "image_url",
ImageURL: &llmrouter.ImageURL{
URL: "data:image/png;base64," + base64.StdEncoding.EncodeToString(imageBytes),
},
},
},
}Document is the equivalent part type for PDFs and similar, {Base64, MediaType}, and it’s Gemini that handles document content natively among the first-class providers. As covered in Part 2, Sarvam only reliably handles a single plain-text part, don’t route multimodal requests there.
Complete Example: Tool Loop, Then Stream the Answer
Tool-call responses are structured JSON the model produces internally, there’s not much value in streaming them. The final, user-facing answer is where streaming actually helps. A practical pattern is non-streaming for the tool loop and streaming only for the last turn:
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"time"
"github.com/bluefunda/llmrouter"
"github.com/bluefunda/llmrouter/middleware"
"github.com/bluefunda/llmrouter/providers/anthropic"
"github.com/bluefunda/llmrouter/providers/openai"
)
var weatherTool = llmrouter.Tool{
Type: "function",
Function: llmrouter.Function{
Name: "get_weather",
Description: "Get current weather for a city",
Parameters: json.RawMessage(`{
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"]
}`),
},
}
func main() {
cb := middleware.NewCircuitBreaker(5, 30*time.Second)
router := llmrouter.New(
llmrouter.WithProvider("openai", openai.NewFromEnv("openai", "OPENAI_API_KEY")),
llmrouter.WithProvider("anthropic", anthropic.NewFromEnv()),
llmrouter.WithFallback("anthropic", "openai"),
llmrouter.WithMiddleware(
middleware.Retry(3, time.Second),
cb.Wrap,
middleware.Timeout(60*time.Second),
),
)
ctx := context.Background()
messages := []llmrouter.Message{
{Role: llmrouter.RoleUser, Content: "What's the weather in Tokyo?"},
}
for {
resp, err := router.Complete(ctx, &llmrouter.Request{
Model: "claude-sonnet-5",
Messages: messages,
Tools: []llmrouter.Tool{weatherTool},
})
if err != nil {
log.Fatal(err)
}
choice := resp.Choices[0]
messages = append(messages, *choice.Message)
if choice.FinishReason != "tool_calls" {
streamFinalAnswer(ctx, router, messages)
if resp.Usage != nil {
fmt.Printf("\n[cost: $%.4f | provider: %s]\n", resp.Usage.Cost, resp.Provider)
}
break
}
for _, tc := range choice.Message.ToolCalls {
var args struct{ City string `json:"city"` }
json.Unmarshal([]byte(tc.Function.Arguments), &args)
// stub result, a real implementation calls a weather API
result := fmt.Sprintf(`{"city": %q, "temp_c": 22, "condition": "sunny"}`, args.City)
messages = append(messages, llmrouter.Message{
Role: llmrouter.RoleTool,
Content: result,
ToolCallID: tc.ID,
})
}
}
}
func streamFinalAnswer(ctx context.Context, router *llmrouter.Router, messages []llmrouter.Message) {
stream, err := router.Stream(ctx, &llmrouter.Request{
Model: "claude-sonnet-5",
Messages: messages,
})
if err != nil {
log.Fatal(err)
}
defer stream.Close()
for stream.Next() {
if event := stream.Event(); event.Type == llmrouter.EventContentDelta {
fmt.Print(event.Content)
}
}
if err := stream.Err(); err != nil {
log.Printf("stream error: %v", err)
}
}A Note on Long Tool Loops
Each iteration of a tool loop appends to the conversation, and every subsequent request re-sends the entire history. Two things worth watching once a loop runs more than a couple of turns:
Context limits. Every model has a maximum context length. A loop with several iterations and verbose tool outputs can get there faster than it looks like it should. Truncate or summarize intermediate tool results before feeding them back if you expect long loops.
Cost accumulation. Every call in the loop pays input-token cost for the entire history up to that point, not just the new turn. A five-iteration tool loop on an expensive model can cost noticeably more than the token count of the final answer alone would suggest. resp.Usage.Cost gives you the cost of each individual call, sum it across the loop for the true cost of the session, covered in full in next Part.
The next part closes the series with cost tracking, prompt caching, and the observability that turns a working integration into one you can actually run in production without surprises: Part 5: Cost Tracking, Prompt Caching, and Production Observability.
Ready to transform your SAP experience?
Join thousands of developers using BlueFunda AI tools.