SALLM, SMALL agents for the LLM world

The name is not cute branding. SALLM stands for SMALL in the agents-for-LLM world.

Not “small” as in toy. Small as in: a bounded context, a predictable prompt, a self-contained process with its own memory, tuned hard for one job. The opposite of the usual agent stack that grows until it needs a product manager, three abstraction layers, and a cloud bill before it can answer “what was the UID of that email?”.

Code lives at https://github.com/codref/sallm-agent - Python, Ollama-friendly, default brain is something like Gemma 4 4B. No DSPy. No Pydantic ceremony. Peewee for SQLite, LanceDB for vectors, LiteLLM to talk to the model. That is roughly the whole shopping list.

Why another agent library

The market is already full of agents. Most of them are excellent at being large.

You get frameworks that want to own your graph, your memory, your tools, your evals, your UI, and eventually your soul. They are powerful. They are also heavy. When the model underneath is a 4B running on a Hetzner box without a GPU (yes, that still works - I wrote about it), the last thing you want is a framework that spends half the tokens explaining itself to itself.

SALLM starts from a different hypothesis:

A small local model stays useful for long sessions if you stop stuffing the entire transcript into every prompt, keep a bounded recent window, retrieve older facts from a durable store, and make the token budget visible.

Raw messages stay canonical. Vectors are a rebuildable index. Derived facts must cite source message ids. When something goes wrong, you get a ContextReceipt that shows what entered the prompt and what was left on the shelf - not an invisible RAG fog.

That is the differentiation in one breath: minimal runtime, durable memory, inspectable budgets, offline-optimized profiles. Not another kitchen-sink agent OS.

Minimal on purpose

A turn looks almost boring, and that is the feature:

user message gets persisted, a tiny control call decides goal / skill / retrieval query, LanceDB retrieves a few memory hits, a budgeted prompt runs a ReAct loop with CLI-shaped tools, the answer is stored, then grounded facts are extracted and indexed.

Tools are not JSON blobs. They look like shell commands (calc --expression "2**10"), they support --help, and intermediate stdout can keep the loop honest. Skills are a stack: default is converse, you register more when a job needs its own prompt fragment and tool subset.

Resume is just reusing state_path + session_id. Kill the process, come back tomorrow, same notebook.

There is also Agent.remember(...) for meaning-first ingest: dump a block of shell history or a briefing, get English facts written for retrieval, without polluting the recent-history window. Useful when the truth is large and ugly, and the questions will be ordinary English later.

Profiles and how we built optimization

Chat never “learns” at startup. That was a deliberate split, and it came from watching other stacks blur two jobs that should stay apart: talking to the model and searching for better prompts.

I looked at DSPy and friends early on. The ideas are good - propose, evaluate, keep winners - but I did not want pickles, compiled graphs, or a second runtime living next to the agent. A SMALL agent should load a boring JSON file and go back to work. So SALLM borrows the shape of prompt search and refuses the framework.

The loop we shipped is intentionally small. A teacher (usually the same Gemma you chat with) rewrites a baseline instruction: clearer, shorter, keep the JSON contract. You score candidates on a tiny JSONL of cases. Successive halvings drop the worse half as the train subset grows, then finalists face the full set. The winner lands in a portable profile: instructions, optional demos, budgets, a dataset fingerprint, seed, and metrics. No DSPy modules in the artifact. Strings and numbers only.

What you optimize, one task at a time: controller (goal / skill / retrieval_query), extractor (grounded facts after a turn), ingest (Agent.remember), converse (extra system guidance), rewriter (standalone retrieval sentence). Empty strings in the packaged gemma4-e4b-v1.json mean “use the built-in baseline”. Non-empty ones replace or supplement it.

Scoring is blunt on purpose. Quality is how many expected fields (or contains needles) you hit. Then soft penalties for tokens and latency, plus a hard fail if a mandatory case misses - so an average score cannot hide a broken skill push. Ten clear controller cases beat a hundred noisy ones. Use the same model you deploy; a bigger teacher invents instructions a 4B cannot follow.

uv run sallm optimize \
  --dataset data/controller_cases.jsonl \
  --task controller \
  --candidates 4 \
  --seed 0 \
  --model ollama/gemma4:e4b-it-qat \
  --out .sallm/profiles/controller-v1.json

--evaluate-only scores the baseline before you burn time on search. Merge winning fields into one profile by hand if you optimize several tasks. Chat only loads --profile; it never searches at startup.

Budgets sit next to the prose: prompt ceiling, recent-history cap, retrieval cap. The receipt after a turn tells you whether you stayed inside them. When history grows for hours, used tokens should hover - not climb with the transcript. Tune wording first, budgets second - otherwise you cannot tell whether a fix came from English or from a larger window.

This matters because a 4B model does not forgive vague system prompts. Optimization is not a luxury add-on; it is how you specialize a SMALL agent without rewriting the framework.

The swarm idea

I do not want one mega-agent that knows email, shell history, tickets, and the meaning of life.

I want thousands of SALLM agents: each self-contained, each with its own SQLite + vector durable memory, each highly specialized and profile-optimized for the task it has to do. Spin one up for an inbox. Spin one up for a histfile. Spin one up for a lab notebook. They are cheap processes, not platforms. If one dies, you restart it on the same paths. If one needs a different tool set, you give it a different skill registry - you do not fork the universe.

That is SMALL as architecture, not only as model size.

A use case, walked through

Imagine a long lab session on a local Gemma.

Early on you say: Please remember this fact for later: the unique lab code is ZEBRA-7711.

Then you chat about clouds, pasta, constellations - filler that pushes the early turn out of the recent-history window. Much later: What is the unique lab code? Reply with the code only.

Without durable memory, a truncated prompt often loses ZEBRA-7711. The model guesses, or politely invents something that looks like a lab code.

With SALLM (--state-path, LanceDB, memory gate on), that early turn was chunked, embedded, and searchable. Control emits a retrieval query, the chunk comes back under the retrieval budget, and Gemma answers from the notebook - not from hope.

You can watch it:

uv run sallm chat \
  --state-path .sallm/state.db \
  --vector-path .sallm/vectors \
  --session lab1 \
  --retrieval-query instruct \
  --search dense \
  --memory-gate \
  --tools none

After a few turns, /context and /memory are the debugging surface. Empty retrieved on the receipt is an honest failure mode - better than a confident hallucination with no trail.

The two examples we ship

IMAP inbox

examples/imap_inbox/ is a durable agent over a live mailbox.

CLI tools list folders, search, fetch bodies. The session is meant to run for many turns: discover mail, anchor short facts, drown the window in filler chatter, then recall From / Subject / UID without re-fetching. Later you can dig into one body, fill again, and still recall a detail from memory.

The point is not “email RAG”. The point is: hours of wall-clock Q&A about a real inbox, prompt size stays predictable, early facts survive eviction, and the same --session resumes after restart. Observability (Tempo + Prometheus + Grafana) is wired in the example so you can see extract latency and retrieval misses instead of arguing with vibes.

Linux shell history

examples/linux_history/ does the same pattern over bash/zsh history, plus a sharper twist: usage-story inference.

Search the histfile with time windows and nearby context. Ingest blocks with meaning-first remember so ordinary-English questions can hit hosts, IPs, and patterns later. Ask something like: around the time I was authenticating on docker, what remote host did I use? The agent reconstructs a short story from grounded hits - without dumping the whole histfile into the model.

There is also bash_run for live commands. That is useful and dangerous in the honest sense: it executes as your user. Treat it like a sharp knife, not a demo toy.

Together the two examples say the same thing in two dialects: specialized agents, durable notebooks, long sessions that do not melt the context window.

Telemetry and tracing

A SMALL agent without eyes is just vibes with a SQLite file. The examples (and sallm chat) can emit OTLP spans and Prometheus metrics without dragging in the OpenTelemetry SDK - a thin tracer plus an in-process metrics server is enough.

Local stack is three containers from the repo compose file: Tempo for spans, Prometheus scraping :9464/metrics, Grafana with a provisioned sallm session dashboard.

docker compose up -d

uv run sallm chat \
  --otlp http://localhost:4318 \
  --metrics-port 9464 \
  --state-path .sallm/state.db \
  --vector-path .sallm/vectors \
  --session demo1

--session is the shared id for Agent state, Prometheus labels, and Tempo session.id. One name, three surfaces - that used to diverge and it was painful.

Each ask() is a root span with children for control, ReAct chat, extract, and tool <name>. Agent.remember() gets its own root. Span attributes carry stack path, goal, active skill, receipt token sections, control action - the same story /context tells, but queryable after the fact.

The dashboard is where the hypothesis becomes visible. Filter by session_id, then walk the rows: session overview (turns, skill, stack depth, retrieval hits, omitted history), token economy (system / retrieval / history composition), control and tools, extract vs queue lens, remember/ingest cost.

Grafana: session overview (turns, skill, stack, retrieval)

Grafana: token economy (prompt composition over turns)

Tempo: one ask() waterfall (control / chat / extract / tools)

Grafana: extract / queue lens (waterfall vs deferred extract)

The extract row earns its keep when you choose --extract waterfall versus queue. Prefer queue when extract latency dominates the turn and miss-flushes stay rare. Prefer waterfall when deferred extract hurts recall turns or the queue depth climbs. The panels exist so you compare the same session under both flags instead of guessing.

Traces inside Tempo are ephemeral unless you add a volume. Metrics live only while the process is up. That is fine for lab work: spin the stack, run a scripted session, screenshot the story, tear down.

What SALLM is not

It will not guarantee the model never invents a fact. Retrieval improves grounding; the receipt makes misses inspectable. It will not replace LangChain if you need an enterprise kitchen. It will not auto-optimize while you chat.

It will sit next to a small local model, keep memory on disk, load a profile you tuned offline, and do one job well - then do it again a thousand times as a thousand small agents.

If that sounds like your kind of pizza (simple ingredients, assembled on purpose), start here: https://github.com/codref/sallm-agent