Introduction

Kova is an async-first Rust library for building LLM-powered agents. An agent, here, is a program that sends messages to a large language model, lets the model call tools you provide, feeds the results back, and repeats until the model produces a final answer. Kova owns that loop so you don’t have to.

The library is deliberately small and orthogonal. Each concern — talking to a model, running a tool, streaming output, embedding text — is a trait with one job. You compose them through a single Agent, and swap any piece without rewriting the rest.

What you get

• A provider-agnostic core. The LlmProvider trait has five first-party implementations — native Anthropic, OpenAI-compatible, AWS Bedrock, Google Gemini, and Ollama. Your agent code doesn’t know or care which one it’s talking to.
• Tool calling, handled for you. Implement the Tool trait, register it, and the agent runs the full call → execute → observe loop, executing multiple tool calls concurrently.
• Stateless by design. Agent::run takes your conversation history in and hands back everything the turn produced — you own persistence, sessions, and scaling. Nothing is read from or written to hidden state.
• Structured output. Constrain a turn’s final text to a JSON schema and parse it straight into your own type with run_structured::<T>(), mapped natively per provider.
• First-class streaming. A pull-based Stream of text, tool-call, and chain-of-thought events — identical across every provider.
• Prompt caching & embeddings. Automatic prompt caching on Anthropic and Bedrock (with per-turn cache-token accounting), plus an EmbeddingProvider trait as the input seam for host-side retrieval.
• MCP integration. Connect Model Context Protocol servers — with automatic reconnect — and their tools become native Tools.
• Reliability and observability. Automatic retries with backoff, classified provider errors, context-budget guards, and CancellationToken-driven turn cancellation, plus optional OpenTelemetry tracing and a metrics collector.

Who it’s for

Kova is for Rust developers building anything that needs an LLM in the loop: chatbots, coding assistants, research agents, data-extraction pipelines, background automation, or multi-agent systems. If you want type safety, async throughout, and the freedom to change model providers without a rewrite, Kova is built for you.

You do not need to be an ML engineer. If you can write an async fn and implement a trait, you can build a capable agent.

The mental model

Everything in Kova hangs off one picture. Keep it in your head as you read the rest of the docs:

   history ──▶  ┌──────────────────────────────┐  ──▶ new_messages
   (you own)    │            Agent             │      (you persist)
                │  ┌──────────┐  ┌───────────┐  │
 swap provider ▶│  │ Provider │  │   Tools   │  │ ◀── you write these
                │  └──────────┘  └───────────┘  │
                └──────────────────────────────┘
                             │ runs
                             ▼
                the agentic loop: ask the model,
                run the tools it asks for, repeat

• The Agent is the orchestration loop. You build one with AgentBuilder.
• A Provider (LlmProvider) is how the agent talks to a model. Required.
• Tools are functions the model can call. Optional but where the power is.
• History is a plain Vec<ConversationMessage> you pass into run and get back (as new_messages) to persist. The agent holds no conversation state — sessions and storage are yours to own.
• Streaming delivers output token-by-token via the run_stream event stream. Optional.

Read The Agentic Loop next — it’s the one concept everything else builds on. Or jump straight to the Quick Start to see it run.

A note on Rust editions and async

Kova targets Rust 2024 edition and is built on Tokio. Every network operation is async; you’ll run agents inside a #[tokio::main] runtime (or any Tokio executor). Traits that cross .await points use async_trait, which you’ll see in the tool and provider examples.