tiny_agent

header-only · C++20 · MIT

Agents that fit
on the board.

tiny_agent is a header-only C++20 framework for tool-using agents. Point it at a frontier API or at a llama.cpp server on a Raspberry Pi, and the agent code does not change.

Read the quickstart View on GitHub 

#include <tiny_agent/tiny_agent.hpp>
#include <tiny_agent/providers/openai.hpp>

using namespace tiny_agent;

auto llm = OpenAIChat{.model="gpt-4o-mini", .api_key=key};

auto response = llm.chat({
    Message::system(
        "You are a concise assistant. Reply in one sentence."),
    Message::user("What is the capital of Japan?")
});

std::cout << response.message.text() << "\n";

Both files ship in examples/. The second one needs no API key: local::ollama(), local::llamacpp() and local::vllm() talk to any OpenAI-compatible server on the machine or the LAN.

stripped binary

7.4MB

17_streaming, Release, Raspberry Pi 5

peak client RSS

2.0MB

during an active streamed response, Pi 5

generation, CPU only

5.48tok/s

Qwen2.5-3B-Instruct Q4_K_M, Pi 5

generation, GPU

23.85tok/s

same model, 36 layers offloaded, Jetson Orin Nano

Measured 2026-08-21 on the named devices, not estimated. Commands and raw output in docs/benchmarks.md.

why edge

The constraints are hard,
not preferences

Inference in C++ is already solved. llama.cpp and Ollama run the model, and llama.cpp's own docs say the agent loop is the caller's problem. That loop is what tiny_agent is: the delegation, the middleware, the tools, the memory, in a few thousand lines of headers with no virtual dispatch anywhere in the dispatch path.

On a device, three things stop being negotiable.

  • offline

    Point local::llamacpp(), local::ollama() or local::vllm() at a server on the machine or the LAN and nothing leaves it. The offline test suite runs the same way: 383 cases, no network, no keys.

  • audit

    Tracing goes to Arize Phoenix, Langfuse, or any OTLP collector, with no vendor SDK in the build. Six log levels, default warn, and trace prints raw HTTP bodies, JSON-RPC messages, tool arguments and results.

  • budget

    7.4 MB of stripped binary and 2.0 MB of RSS on a Pi 5, measured while streaming. The agent layer is not the cost. The model is.

what it does

What ships in the headers

DeepAgent: sub-agents are just tools

agent_as_tool() wraps a shared_ptr<Agent> in a DynamicTool with a JSON schema, so delegation needs no new concept and no depth limit. Each sub-agent can run a different model from a different provider.

A tool that throws never takes down the loop: the exception comes back as a tool_result the model reads and recovers from, and truncated tool arguments get the same treatment instead of an exception out of the JSON parser.

director gpt-4o analyst claude-sonnet-4-5 fact_checker gpt-4o-mini

The tree from the README's DeepAgent snippet: three agents, two providers, one run(). Working versions in examples/06_deep_agent.cpp and 16_deep_agent_custom.cpp.

Middleware, one signature

A middleware takes the mutable message vector and a Next it must call to reach the model. That is enough to rewrite the prompt, short-circuit without calling the model, retry by calling next twice, or fall back to another provider. Fourteen ship built in. make_middleware_stack() folds the same callables through a std::tuple with no std::function and no allocation.

Providers behind concepts

There is no IChatModel. A provider is a full specialization of LLMModel<Provider, Kind> that satisfies a concept, so the model call is direct: no vtable, no type erasure. Seven ship, and adding one is a header rather than a change to the core.

  • OpenAI
  • Anthropic
  • Gemini
  • Mistral
  • Cohere
  • VoyageAI
  • Ollama
  • llama.cpp
  • vLLM

MCP, both transports

stdio and HTTP, with no paid tier gating either one. mcp::connect_stdio() and the HTTP client discover tools from a running server and hand them to an agent the same way as any other tool.

Real SSE streaming

Token deltas off the wire, tool-call deltas included, on the OpenAI-compatible and Anthropic paths. run_stream() drives the full ReAct loop while the text arrives. Streaming is a separate concept refinement, so providers without it still satisfy is_chat and the agent branches with if constexpr.

Retrieval and vector stores

One four-method concept behind all seven backends: Flat and hnswlib in process, Qdrant, Chroma, Weaviate, Redis and Milvus over the network. Embeddings and batch are in the box.

Tracing and Agent Skills

Spans go to Arize Phoenix, Langfuse, or any OTLP collector, with no vendor SDK in the build. Agent Skills load from SKILL.md, which no other C++ framework does.

Also shipping: context management against an explicit token budget, multimodal messages, init_chat_model("openai:gpt-4o") for provider selection at runtime, and Runnable composition with operator|.

measured, not estimated

On the actual hardware

Same source tree, same model, same quantization, three machines. Every figure below came off a device on 2026-08-21 with the commands recorded in the repo.

Metric Raspberry Pi 516GB, aarch64, CPU Jetson Orin Nano Super8GB, aarch64, CUDA macOS arm64M-series
Streaming example, stripped Release binary 7.4 MB7.6 MB7.7 MB
Peak client RSS during a streamed response 2.0 MB5.1 MBnot measured
Generation speed, Qwen2.5-3B-Instruct Q4_K_M 5.48 tok/s23.85 tok/snot measured
Prompt eval speed not reported874.77 tok/snot measured
Offline ctest suite, on device 20/20 0.06 s20/20 0.16 snot measured
Clean full Release build 647 s524 snot measured

The Jetson's 5.1 MB against the Pi's 2.0 MB is the toolchain, not the workload: g++ 11.4 on JetPack 6.2 there, g++ 14.2 on Debian 13 on the Pi. Peak RSS held at 5.0 to 5.2 MB across five Jetson runs whether the GPU was doing the work or not. Pi prompt-eval is left out of this run because the fixed prompt hit the server's KV cache and the number would have flattered it.

Agent layer, on the Pi 5

What the orchestration itself costs, from bench_agent on the same board.

Tool lookup, 20 tools130 ns7.2M ops/s
Static middleware chain, 5 deep74 ns12.4M ops/s
Runtime middleware chain, 5 deep667 ns1.5M ops/s
Agent run, full stack9.4 µs105.7K ops/s
Static versus runtime middleware, depth 1026.3 ×compile-time chain

Full methodology, the raw bench_agent output and the on-device bootstrap are in docs/benchmarks.md, with per-device proofs for the Pi 5 and the Jetson.

quickstart

Five lines to a running agent

You need CMake 3.20 or newer, a C++20 compiler, and vcpkg with VCPKG_ROOT set. Configuring installs nlohmann-json, cpp-httplib with OpenSSL, doctest, libenvpp and json-schema-validator. The library target itself only needs the first two; the rest are for tests.

On Windows PowerShell the same commands work with $env:VCPKG_ROOT = "C:\src\vcpkg" and --config Debug on the build and test steps.

build and run 
export VCPKG_ROOT="$HOME/src/vcpkg"
export OPENAI_API_KEY="your-key-here"

cmake --preset default          # or --preset release
cmake --build --preset default
./build/examples/01_basic_chat
vendor it into your build 
add_subdirectory(external/tiny_agent_cpp)
target_link_libraries(my_app PRIVATE tiny_agent)

Prefer an install? cmake --install and find_package(tiny_agent CONFIG REQUIRED) work too, and ports/tiny-agent in the repo is a working vcpkg overlay port until it upstreams. Set TINY_AGENT_BUILD_EXAMPLES, TINY_AGENT_BUILD_TESTS and TINY_AGENT_BUILD_BENCH to OFF for a vendored build.

faq

Questions people actually ask

Is it really header-only, and what are the dependencies?

Yes. #include <tiny_agent/tiny_agent.hpp> and you are building against headers, with no separate library step. The library itself needs nlohmann-json and cpp-httplib with OpenSSL; doctest, libenvpp and json-schema-validator are test-only. TINY_AGENT_HNSWLIB pulls in hnswlib if you want that vector store, off by default.

Does it run fully offline and local?

Against a local server, yes. Point local::llamacpp(), local::ollama() or local::vllm() at a server on your machine or LAN and nothing leaves it. The offline test suite runs the same way, 383 cases with no network and no keys. Cloud providers need a connection, obviously; the agent code does not change either way.

How small is it really?

A complete streaming agent example is a 7.7 MB stripped binary on macOS arm64 with TLS included, and 7.4 to 7.6 MB on a Raspberry Pi 5 or a Jetson Orin Nano. Client RSS while streaming from a local llama.cpp server measured 2.0 MB on the Pi 5.

How do agents call other agents?

agent_as_tool() wraps a shared_ptr<Agent> in a DynamicTool with a JSON schema, so a director agent calls an analyst agent the same way it calls any function, and the analyst can call a fact-checker in turn. Each sub-agent can run a different model from a different provider. That is the whole mechanism behind DeepAgent: no separate delegation concept, just a tool.

What C++ standard and compilers does it need?

C++20, set by CMAKE_CXX_STANDARD 20 in the build. CI compiles and runs the test suite on every push across Linux x64 and arm64, macOS arm64, and Windows x64. Compilers as old as g++ 11.4 are covered: the one known overload ambiguity it hit is fixed and pinned by a regression test.

What is the license?

MIT, copyright Riadh Haj Amor. Vendor it, link it, modify it, ship it inside a closed-source or commercial product, all without asking. The only obligation is keeping the copyright notice and license text with the code you took it from.

The rest of the FAQ, the provider list and the platform notes live in the README.