Configuring a Provider

import { Tabs, TabItem } from ‘@astrojs/starlight/components’;

A provider is the only model-specific code in your agent (see the Providers concept). This guide is the practical how-to for each of the five first-party providers. Build one, wrap it in Arc, and pass it to AgentBuilder::provider — the rest of your code is identical whichever you pick.

The anthropic feature speaks the native Messages API (POST /v1/messages) — streaming SSE, tool use, adaptive extended thinking, and automatic prompt caching, all on by default.

use std::sync::Arc;
use kova_sdk::provider::anthropic::{AnthropicProvider, AnthropicProviderConfig};

let config = AnthropicProviderConfig::new("claude-opus-4-8")
    .with_api_key(std::env::var("ANTHROPIC_API_KEY").expect("set ANTHROPIC_API_KEY"));

let provider = Arc::new(AnthropicProvider::new(config)?);

Reasoning effort and thinking knobs:

let config = AnthropicProviderConfig::new("claude-opus-4-8")
    .with_api_key("sk-ant-…")
    .with_effort("high")             // output_config.effort: low|medium|high|xhigh|max
    .with_adaptive_thinking(true);   // thinking: {"type":"adaptive"} — on by default

Prompt caching is on by default — a top-level cache_control: {"type": "ephemeral"} marks the last cacheable block, so the system prompt, tools, and prior turns are served from Anthropic’s cache on repeat turns. Reads/writes come back as UsageStats::cache_read_tokens / cache_creation_tokens. Turn it off with .with_cache(false). The Messages API requires max_tokens; Kova defaults to 32k, override with .with_default_max_tokens(n).

Field	Default	Description
`model`	required	e.g. `"claude-opus-4-8"`
`api_key`	`None`	Sent as `x-api-key`
`base_url`	Anthropic API	Override endpoint
`timeout`	60s	Request timeout
`default_max_tokens`	32000	Messages API requires `max_tokens`
`adaptive_thinking`	`true`	Sends `thinking: {"type":"adaptive"}`
`effort`	`None`	`output_config.effort`: `low`/`medium`/`high`/`xhigh`/`max`
`cache`	`true`	Automatic ephemeral prompt caching

Signed thinking blocks round-trip through history as ContentBlock::Thinking — required for tool loops with extended thinking (Kova handles it as long as you persist new_messages verbatim).

The openai feature works with hosted OpenAI, Azure OpenAI, vLLM, LM Studio, LiteLLM, and local Ollama’s OpenAI endpoint — anything speaking the Chat Completions API.

use std::sync::Arc;
use std::time::Duration;
use kova_sdk::provider::openai::{OpenAiCompatibleProvider, OpenAiProviderConfig};

let config = OpenAiProviderConfig::new("https://api.openai.com", "gpt-4o")
    .with_api_key("sk-…")
    .with_timeout(Duration::from_secs(60))
    .with_max_tokens(4096)
    .with_temperature(0.7);

let provider = Arc::new(OpenAiCompatibleProvider::new(config)?);

Hosted OpenAI o-series (reasoning):

let config = OpenAiProviderConfig::new("https://api.openai.com", "o3")
    .with_api_key("sk-…")
    .with_reasoning_effort("high");   // "low" | "medium" | "high"

Azure OpenAI — override the endpoint paths and API version:

let config = OpenAiProviderConfig::new("https://my-resource.openai.azure.com", "gpt-4")
    .with_api_key("…")
    .with_api_version("2024-02-01")
    .with_chat_completions_path("/openai/deployments/gpt-4/chat/completions")
    .with_models_path("/openai/deployments/models");

Local server (LM Studio, vLLM) — no API key:

let config = OpenAiProviderConfig::new("http://127.0.0.1:1234", "my-model");

Field	Default	Description
`base_url`	required	API base URL
`model`	required	Model identifier
`api_key`	`None`	Bearer token
`timeout`	30s	Request timeout
`max_tokens`	`None`	Max completion tokens
`temperature`	`None`	Sampling temperature
`chat_completions_path`	`/v1/chat/completions`	Chat endpoint path
`models_path`	`/v1/models`	Models-list endpoint path
`api_version`	`None`	Query param (e.g. Azure `api-version`)
`reasoning_effort`	`None`	`"low"`/`"medium"`/`"high"` — o-series only

The bedrock feature uses the Bedrock Converse / ConverseStream APIs with SigV4 signing. It pulls in the AWS SDK crates — if you don’t need it, disable default features (see Installation).

use kova_sdk::provider::bedrock::{BedrockProvider, BedrockProviderConfig};

// Default AWS credential chain (env vars, ~/.aws/credentials, IAM role, …)
let config = BedrockProviderConfig::new(
    "us-east-1",
    "anthropic.claude-sonnet-4-20250514-v1:0",
);
let provider = Arc::new(BedrockProvider::new(config).await?);   // note: async

Credentials resolve in this order: explicit → named profile → default chain.

// Named profile
let config = BedrockProviderConfig::new("us-west-2", "anthropic.claude-sonnet-4-20250514-v1:0")
    .with_profile("my-profile");

// Explicit credentials
let config = BedrockProviderConfig::new("us-east-1", "anthropic.claude-sonnet-4-20250514-v1:0")
    .with_credentials("AKIA…", "secret", Some("session-token".into()));

// Custom endpoint (LocalStack, etc.)
let config = BedrockProviderConfig::new("us-east-1", "my-model")
    .with_endpoint_url("http://localhost:4566");

Extended thinking on Claude models — pass model-specific fields through:

let config = BedrockProviderConfig::new("us-east-1", "anthropic.claude-sonnet-4-20250514-v1:0")
    .with_additional_model_request_fields(serde_json::json!({ "budgetTokens": 5000 }));

BedrockProvider::new is async (it resolves credentials once at construction). Credentials are fixed for the provider’s lifetime — rebuild it to rotate them.

Prompt caching (opt-in) places a cachePoint after the system prompt and after the last message, mirroring the native Anthropic provider’s placement. Only cachePoint-capable models (Anthropic Claude, Amazon Nova) accept it:

let config = BedrockProviderConfig::new("us-east-1", "anthropic.claude-sonnet-4-20250514-v1:0")
    .with_cache(true);   // off by default; cache-usage fields are read regardless

Field	Default	Description
`region`	required	AWS region
`model_id`	required	Bedrock model id
`profile`	`None`	AWS named profile
`access_key_id` / `secret_access_key` / `session_token`	`None`	Explicit creds
`timeout`	60s	Request timeout
`endpoint_url`	`None`	Override endpoint
`cache`	`false`	Adds `cachePoint`s (cachePoint-capable models only)
`additional_model_request_fields`	`None`	Arbitrary JSON → `additionalModelRequestFields`

The gemini feature targets generativelanguage.googleapis.com, authenticating with an x-goog-api-key header.

use std::time::Duration;
use kova_sdk::provider::gemini::{GeminiProvider, GeminiProviderConfig};

let config = GeminiProviderConfig::new("gemini-2.0-flash")
    .with_api_key("AIza…")
    .with_timeout(Duration::from_secs(60));

let provider = Arc::new(GeminiProvider::new(config)?);

Extended thinking on thinking-capable models:

// -1 = dynamic/unlimited, 0 = off (default), positive = token cap
let config = GeminiProviderConfig::new("gemini-2.5-flash")
    .with_api_key("AIza…")
    .with_thinking_budget(5000);

Testing against a mock server / future API versions:

let config = GeminiProviderConfig::new("gemini-2.0-flash")
    .with_base_url("http://localhost:8080")
    .with_api_version("v1beta");

Field	Default	Description
`model`	required	e.g. `"gemini-2.0-flash"`
`api_key`	`None`	Sent as `x-goog-api-key`
`timeout`	60s	Request timeout
`base_url`	`generativelanguage.googleapis.com`	API base URL
`api_version`	`"v1beta"`	Path segment before `/models/`
`thinking_budget`	`None` (off)	`-1` unlimited, `0` off, positive = cap

Streaming uses ?alt=sse so it reuses the shared SSE parser; chain-of-thought parts (thought: true) are filtered from visible content into ModelResponse::thinking.

The ollama feature talks to a local (or remote) Ollama server. No API key required.

use std::time::Duration;
use kova_sdk::provider::ollama::{OllamaProvider, OllamaProviderConfig, OllamaThink};

// Local, default http://localhost:11434
let config = OllamaProviderConfig::new("llama3.2");
let provider = Arc::new(OllamaProvider::new(config)?);

// Remote instance with a longer timeout
let config = OllamaProviderConfig::new("llama3.2")
    .with_base_url("http://my-server:11434")
    .with_timeout(Duration::from_secs(180))
    .with_keep_alive("10m");

Thinking-capable model (qwen3, deepseek-r1, …):

let config = OllamaProviderConfig::new("qwen3")
    .with_think(OllamaThink::High);   // Enabled | High | Medium | Low

Extra generation options (merged into Ollama’s options):

use serde_json::json;
let mut opts = serde_json::Map::new();
opts.insert("temperature".into(), json!(0.7));
opts.insert("num_ctx".into(), json!(8192));
let config = OllamaProviderConfig::new("llama3.2").with_extra_options(opts);

Field	Default	Description
`model`	required	e.g. `"llama3.2"`, `"qwen3"`
`base_url`	`http://localhost:11434`	Server URL
`timeout`	120s	Local inference can be slow
`keep_alive`	`None`	e.g. `"5m"`, or `"0"` to unload immediately
`think`	`None`	`OllamaThink::*` — needs a thinking model
`extra_options`	`None`	Merged into `options` (`top_k`, `seed`, `num_ctx`, …)

Streaming uses NDJSON over /api/chat; list_models uses /api/tags. Tool calls are supported.

Swapping providers

Because everything hangs off the LlmProvider trait, switching is a build-site-only change. This function is provider-agnostic:

fn make_agent(provider: Arc<dyn LlmProvider>) -> Result<Agent, KovaError> {
    AgentBuilder::new()
        .provider(provider)
        .system_prompt("You are a helpful assistant.")
        .tool(Arc::new(GetWeather))
        .build()
}

// Choose the model at the edges of your program:
let provider: Arc<dyn LlmProvider> = match std::env::var("PROVIDER").as_deref() {
    Ok("anthropic") => Arc::new(AnthropicProvider::new(anthropic_cfg)?),
    Ok("bedrock")   => Arc::new(BedrockProvider::new(bedrock_cfg).await?),
    Ok("ollama")    => Arc::new(OllamaProvider::new(ollama_cfg)?),
    _               => Arc::new(OpenAiCompatibleProvider::new(openai_cfg)?),
};
let agent = make_agent(provider)?;

Tuning inference

Model parameters that aren’t provider-specific go in InferenceConfig on the builder, so they apply regardless of provider:

let agent = AgentBuilder::new()
    .provider(provider)
    .inference_config(InferenceConfig {
        model: Some("gpt-4o".into()),
        max_tokens: Some(2048),
        temperature: Some(0.2),
        top_p: None,
        stop_sequences: Some(vec!["\n\nUser:".into()]),
        ..Default::default()   // response_format defaults to None
    })
    .build()?;

Override per turn with agent.run_with_config(&history, overrides) — unset fields fall back to the agent’s config. The remaining field, response_format, is covered in Structured Output.

Providers concept — the trait and mental model, plus writing a custom provider.
Thinking & Reasoning Models — using the reasoning knobs shown above.
Feature Flags — trimming the AWS dependency tree.

Configuring a Provider

Swapping providers

Tuning inference

Related