Programmable Databases: Why We Built liath Twice

This article was rewritten in October 2026: an earlier version misdescribed both implementations and quoted performance comparisons that had not been measured.

We built the same database twice. Not because the first one was a mistake, but because the question it was built to explore — what happens when an agent can write programs against its own memory — turned out to need a different foundation than the one we started with. liath is a Python package that embeds Lua. liath-rs is a Rust crate that embeds Lua. They share an idea and a query language, and they differ in almost everything underneath.

This is the story of why we built them, what changed between them, and what we would have to measure before saying anything about which is faster.

Why a Programmable Database

The motivation comes from a specific pain point in agent design. Most agent memory systems expose a fixed API: semantic_search(query, k), perhaps with a metadata filter. The retrieval strategy is decided by whoever wrote the API, not by the agent that needs the memory.

But retrieval strategies vary by task. One task wants the twenty most similar memories filtered to the last week. Another wants anything tagged with a project name, ranked by importance. A third wants to cross-reference two collections. A fixed API either grows an endless list of parameters or forces the application to pull everything back and filter it in application code.

liath’s answer is to make the query language a real programming language. An agent — or an LLM generating code on the agent’s behalf — writes a small Lua program, and the database runs it next to the data. Lua is a deliberate choice: it is small, it embeds cleanly, it is easy for models to generate, and its standard library can be cut down to almost nothing, which matters when the author of the query is a model.

In liath-rs the storage primitives are available directly inside Lua:

-- Key-value storage
put("config", "theme", "dark")
local theme = get("config", "theme")

-- Semantic storage: text is embedded and indexed automatically
store_with_embedding("notes", "n1", "Meeting notes from Monday")
local relevant = semantic_search("notes", "weekly meetings", 5)

Everything after the retrieval — filtering by recency, re-scoring by importance, joining against another namespace — is ordinary Lua, written for the task in hand.

liath v1: A Python Package That Speaks Lua

The first implementation is a Python package. Python hosts the database; Lua is embedded through lupa; storage defaults to LevelDB through plyvel, with RocksDB available as an optional backend for heavier use. AI capabilities arrive as plugins installed through pip extras: embed for FastEmbed embeddings, vdb for USearch vector search, llm for model access. Namespaces isolate data for multi-tenant use, and the package ships both a CLI and an HTTP server.

From Python, using it looks like this:

from liath import EmbeddedLiath

db = EmbeddedLiath(data_dir="./data")
db.put("user:1", '{"name": "Alice", "role": "admin"}')

result = db.execute_lua('''
    local user = db:get("user:1")
    return "Found: " .. user
''')

db.close()

What Worked

Rapid iteration. Python is where most agent code already lives, and pip extras made it easy to bolt on embeddings, vector search and model access one at a time. For exploring what a programmable memory should even look like, that velocity mattered more than anything else.

The query model. Letting Lua run next to the data held up. The interesting queries — the ones a fixed API cannot express — turned out to be short programs, and short programs are something models write reliably.

Embeddability. There is no separate server to start unless you want one. You construct EmbeddedLiath and you have a database.

What Pushed Us to Rebuild

Three runtimes in one process. A query in v1 crosses from Python into Lua and back into Python-wrapped native storage libraries. Each boundary is a place where data is converted, where errors have to be translated, and where reasoning about cost gets harder.

Dependency weight. The full feature set means Python, a Lua runtime, native LevelDB or RocksDB bindings, an embedding library and a vector index, each with its own build story. “A single dependency” is not what a user experiences.

A sandbox in a permissive host. v1 includes a file plugin for reading and writing files. That is useful for a developer and the wrong default if the query author is a model. The safety story we wanted — a model can write any query it likes and still cannot touch the host — needed to be designed in from the start, not configured afterwards.

Reach. A Python package is only usable from Python. We wanted an engine that could be embedded in a Rust service as a library, and also run as a terminal console, an HTTP server or an MCP server, from one binary.

liath-rs: The Rust Rebuild

liath-rs keeps the idea and replaces the foundation. It is a Rust crate that embeds Lua through rlua, stores key-value data in Fjall, indexes vectors with USearch and generates embeddings with FastEmbed. Cargo features select the interfaces: a terminal UI (on by default), an HTTP API server and an MCP server for AI assistants (both off by default). Python users are still pointed at the separate Python package, so for now there are genuinely two implementations to maintain.

The storage choice is worth a sentence. Fjall is an LSM-tree key-value store written in Rust. Choosing it over RocksDB keeps the whole build in one language and toolchain, with no C++ library to compile and link — which is what “single dependency” has to mean in practice.

The Lua surface also changed. In v1, queries call methods on a db object (db:get("user:1")). In liath-rs, queries call namespaced primitives directly (get("config", "theme")), and vector and semantic search are first-class functions rather than plugin calls. The README describes the Lua runtime as sandboxed: no file system, no network, no system calls. That is the property that makes it reasonable to let an LLM author queries, and it is a claim to verify rather than accept — we have not yet published an independent audit of the sandbox.

What We Would Measure, and How

An earlier version of this article quoted latency and memory comparisons between the two implementations. We have withdrawn them: they had not been measured to a standard we would defend. What we can do is set out what a fair comparison would require, because it is less simple than it looks.

Control the query surface. The two implementations do not expose identical Lua APIs, so a comparison must run the same logical workload through a thin shim on each side and report the shim’s cost. Otherwise a difference in API design masquerades as a difference in engine.

Separate the variables. v1 and liath-rs differ in host language, storage engine and Lua binding at once. A result that says “liath-rs is faster” without isolating which of those three changes produced it is an anecdote. The cleanest design runs v1 on both of its backends (LevelDB and RocksDB) and liath-rs on Fjall, with the same data, so at least the storage contribution can be bounded.

Measure the tail, not the mean. Write latency at a fixed offered load, reported as p50, p99 and p99.9 over sustained runs, with warm-up excluded. Point reads and full scans with a Lua predicate separately, because the bottleneck moves: point reads stress storage, scans stress Lua evaluation.

Measure memory honestly. Resident memory for the same dataset, with the embedding model and vector index reported separately from the key-value store, since they dominate for any realistic agent memory.

Measure the sandbox. A corpus of hostile Lua — attempts to reach io, os, require, debug, load with crafted bytecode, and resource exhaustion through infinite loops or runaway allocation — run against both implementations, reporting what is blocked and what is not.

Until those results exist, the honest statement is architectural: liath-rs removes a language boundary and a native dependency from the query path and was designed around a sandbox from the start. Whether that moves the numbers that matter for your workload is an empirical question.

When to Rewrite vs. When to Optimise

The move from liath to liath-rs was a rewrite, and it is worth being clear about why, because “we rewrote it in Rust” is not an argument.

The reasons were architectural rather than about raw speed: the number of runtimes in the query path, the dependency footprint a user had to install, the need for a sandbox designed in rather than configured, and the wish to serve embedded, console, HTTP and MCP deployments from one engine. None of those can be fixed by tuning the Python implementation. All of them are properties of where the code lives.

Here is the heuristic we would apply to the next one:

• If the bottleneck is algorithmic: optimise the algorithm. The language does not matter.
• If the problem is a property of the runtime — the boundaries a call crosses, the dependencies it drags in, the isolation it can offer — consider rewriting the component that owns that property.
• If you rewrite, decide in advance what you will measure. We did not, and it is why this article can describe the change but not quantify it.

Lessons for AI Infrastructure

Prototype in the comfortable language. Python was the right place to discover what a programmable agent memory should do. The prototype told us what the interface should look like.

Design safety in when the user is a model. A database whose queries are written by an LLM has a different threat model from one whose queries are written by its developer, and that difference belongs in the architecture.

Keep a comparison possible. Changing the host language, the storage engine and the query surface in one step made the rebuild easier to ship and harder to evaluate. If the point of building something twice is to learn from the difference, change fewer things at once — or build the shim that lets you compare like with like.

We encourage researchers working on programmable databases, embedded scripting for data systems or agent memory to try both implementations, and particularly to run the sandbox tests above.