Providers

A provider is Kova’s adapter to a language model. It’s the one piece that knows how to speak a specific model API — building the HTTP request, parsing the response, translating tool calls and streaming chunks into Kova’s own types. Everything above it (the agent, your tools, your history) is provider-agnostic.

The LlmProvider trait

Every provider implements one object-safe async trait:

#[async_trait]
pub trait LlmProvider: Send + Sync {
    // One blocking completion.
    async fn chat_completion(
        &self,
        messages: &[ConversationMessage],
        tools: &[ToolDefinition],
        config: &InferenceConfig,
    ) -> Result<ModelResponse, KovaError>;

    // A stream of incremental events.
    async fn chat_completion_stream(
        &self,
        messages: &[ConversationMessage],
        tools: &[ToolDefinition],
        config: &InferenceConfig,
    ) -> Result<Pin<Box<dyn Stream<Item = Result<StreamEvent, KovaError>> + Send>>, KovaError>;

    // Enumerate available models.
    async fn list_models(&self) -> Result<Vec<ModelInfo>, KovaError>;

    // Count the prompt tokens for a would-be request. Has a default
    // (an offline ~4-chars/token heuristic); Anthropic overrides it with the
    // native /v1/messages/count_tokens endpoint. Powers context budgets.
    async fn count_tokens(&self, messages: &[ConversationMessage], tools: &[ToolDefinition])
        -> Result<u32, KovaError>;
}

Because the agent only depends on this trait (Arc<dyn LlmProvider>), swapping from Anthropic to Bedrock to a local Ollama model is a one-line change where you build the provider. Your tools, loop, and streaming code never change.

All first-party providers are stateless HTTP clients — they hold no mutable per-conversation state. The conversation lives in the caller, which is what makes them cheap to Arc-share across concurrent requests.

The five first-party providers

Provider	Struct	Auth	Feature
Anthropic (native)	AnthropicProvider	x-api-key	anthropic
OpenAI-compatible	OpenAiCompatibleProvider	Bearer token (optional)	openai
AWS Bedrock	BedrockProvider	SigV4 (profile / keys / chain)	bedrock
Google Gemini	GeminiProvider	x-goog-api-key	gemini
Ollama	OllamaProvider	none (local)	ollama

All five are enabled by default. The native Anthropic provider speaks the Messages API directly (streaming SSE, tool use, adaptive extended thinking, and automatic prompt caching); Claude models are also reachable through Bedrock. See Configuring a Provider for the full config surface of each; here’s the shape:

use kova_sdk::provider::anthropic::{AnthropicProvider, AnthropicProviderConfig};

let config = AnthropicProviderConfig::new("claude-opus-4-8")
    .with_api_key(std::env::var("ANTHROPIC_API_KEY")?);
let provider = Arc::new(AnthropicProvider::new(config)?);

let agent = AgentBuilder::new().provider(provider).build()?;

Prompt caching

The Anthropic and Bedrock providers can serve the stable prefix of a conversation (system prompt, tool definitions, prior turns) from the provider’s prompt cache — a large cost and latency win on multi-turn or long-context agents. On Anthropic it’s on by default; on Bedrock it’s opt-in (with_cache(true)). Cache reads and writes are reported per call as UsageStats::cache_read_tokens / cache_creation_tokens, and input_tokens never double-counts cached tokens, so input_tokens + cache_read_tokens is always the full prompt.

OpenAI-compatible covers a lot

The openai provider speaks the OpenAI Chat Completions API, which is a de facto standard. It works with hosted OpenAI, Azure OpenAI, vLLM, LM Studio, LiteLLM proxies, and local Ollama’s OpenAI endpoint — anything that exposes /v1/chat/completions. The base URL and endpoint paths are configurable, so Azure deployments and proxies work without a separate code path.

InferenceConfig: controlling the call

Per-call model parameters live in InferenceConfig, set on the agent once and cloned into every provider call:

let agent = AgentBuilder::new()
    .provider(provider)
    .inference_config(InferenceConfig {
        model: Some("gpt-4o".into()),
        max_tokens: Some(2048),
        temperature: Some(0.3),
        top_p: None,
        stop_sequences: None,
        ..Default::default()
    })
    .build()?;

All fields are Option — unset means “use the provider/model default”. You can also override per turn with run_with_config(messages, overrides), where unset override fields fall back to the agent’s config. InferenceConfig also carries response_format for structured output — hence the ..Default::default() above.

What comes back: ModelResponse

A blocking call returns a ModelResponse:

pub struct ModelResponse {
    pub content: Vec<ContentBlock>,   // text and/or tool-use blocks
    pub stop_reason: StopReason,      // EndTurn | ToolUse | MaxTokens | Unknown
    pub usage: Option<UsageStats>,    // token counts, if the provider reports them
    pub thinking: Option<String>,     // chain-of-thought, if a thinking model
}

You rarely touch ModelResponse directly — the agent consumes it to drive the loop and hands you an AgentResponse. It matters when you implement a custom provider.

Thinking / reasoning models

Reasoning models (Claude extended thinking on native Anthropic and Bedrock, OpenAI o-series, Gemini thinking, Ollama qwen3/deepseek-r1) emit chain-of-thought separately from their answer. Kova extracts the visible reasoning into ModelResponse::thinking (and StreamEvent::ThinkingDelta while streaming), which is never re-sent to the model as prompt.

One exception: Anthropic requires signed thinking blocks to be replayed verbatim when a tool-use turn continues, so Kova round-trips those through history as ContentBlock::Thinking; other providers drop them when building requests. See Thinking & Reasoning Models.

Writing a custom provider

Any model API can join the party — implement LlmProvider:

use kova_sdk::provider::LlmProvider;
use kova_sdk::models::*;
use kova_sdk::error::KovaError;
use async_trait::async_trait;
use std::pin::Pin;
use futures::Stream;

struct MyProvider { /* http client, endpoint, key … */ }

#[async_trait]
impl LlmProvider for MyProvider {
    async fn chat_completion(
        &self,
        messages: &[ConversationMessage],
        tools: &[ToolDefinition],
        config: &InferenceConfig,
    ) -> Result<ModelResponse, KovaError> {
        // 1. translate `messages` + `tools` into your API's request shape
        // 2. POST it
        // 3. map the response back into ModelResponse { content, stop_reason, .. }
        todo!()
    }

    async fn chat_completion_stream(
        &self,
        messages: &[ConversationMessage],
        tools: &[ToolDefinition],
        config: &InferenceConfig,
    ) -> Result<Pin<Box<dyn Stream<Item = Result<StreamEvent, KovaError>> + Send>>, KovaError> {
        todo!()
    }

    async fn list_models(&self) -> Result<Vec<ModelInfo>, KovaError> { todo!() }
}

Wrap it in Arc and pass it to AgentBuilder::provider like any other. The agent can’t tell the difference.

Key takeaways

• A provider is the only model-specific code; everything above it is agnostic.
• Swapping models = swapping the provider you build. Nothing else changes.
• Five providers ship first-party; the native Anthropic provider adds automatic prompt caching, and OpenAI-compatible covers most other hosted and local servers.
• InferenceConfig controls model parameters; ModelResponse is what a call returns.
• Implement LlmProvider to support any other API.