Safe Computing

Programmable Databases: Why We Built liath Twice

The story of building a Lua-native database in Lua, then rewriting it in Rust with RocksDB — and what the performance tells us.

By Dipankar Sarkar
databaseluarustRocksDBstorage-engines

We built the same database twice. Not because we made a mistake the first time — though the first version had plenty of shortcomings — but because the comparison is the point. liath is a Lua-native database written in Lua. liath-rs is the same database rewritten in Rust with RocksDB as the storage engine, exposing the same Lua query interface. Building both gives us something that a single implementation cannot: a controlled experiment on the impact of storage engine choice under identical query semantics.

This is the story of why we built them, what we learned, and what the performance delta tells us about when to rewrite versus when to optimise.

Why a Programmable Database

The motivation for liath comes from a specific pain point in AI workflow prototyping. When you are building an AI pipeline — say, a multi-step agent that retrieves documents, processes them, stores intermediate results, and generates output — you spend a surprising amount of time on the plumbing between your application logic and your data storage.

A typical setup looks like this: your agent logic is in Python (or Lua, or TypeScript). Your data is in PostgreSQL, or Redis, or a collection of JSON files. Between them is a serialisation layer, an ORM or query builder, connection pooling, error handling, and schema migration. This plumbing is not your research contribution. It is overhead.

What if the query language and the application language were the same? What if you could write a database query as a function in the same language your agent logic is written in, with no serialisation boundary, no impedance mismatch, no ORM?

This is what liath provides. It is a database where queries are Lua functions that operate directly on Lua tables. There is no SQL parser, no query planner, no separate query language. You store Lua tables, and you query them with Lua functions.

-- Store a document
db:put("doc:1", {
    title = "Attention Is All You Need",
    year = 2017,
    citations = 120000,
    tags = {"transformers", "attention", "nlp"}
})

-- Query: find all documents from 2017 with more than 10000 citations
local results = db:query(function(key, value)
    return value.year == 2017 and value.citations > 10000
end)

The query function has full access to the Lua runtime. It can call other functions, access closures, perform complex computations. This is dramatically more expressive than SQL for the kind of ad-hoc, prototype-stage queries that AI workflow development requires.

Building liath v1 in Pure Lua

The first version of liath is written entirely in Lua, including the storage engine. Data is stored in a LuaJIT-friendly format using Lua’s built-in serialisation capabilities. The on-disk format is a log-structured merge tree (LSM) implemented in pure Lua, with WAL (write-ahead log) for durability.

What Worked

Rapid iteration. With everything in Lua, the entire stack is modifiable at runtime. During development, we could change the storage format, the indexing strategy, or the query execution model without recompiling anything. For a research prototype, this velocity is invaluable.

Tight integration. Because queries are Lua functions and data is Lua tables, there is zero serialisation overhead between the application and the database. A query receives actual Lua tables, not deserialised copies. For small to medium datasets, this eliminates what is often the dominant cost in database access: marshalling data across a language boundary.

Embeddability. liath runs inside any Lua or LuaJIT process. There is no separate server to start, no socket to connect to, no protocol to speak. You require("liath") and you have a database. For embedding in AI agents that need persistent storage, this is ideal.

What Did Not Work

Performance under load. Pure Lua is not fast enough for a storage engine. LuaJIT helps enormously — its trace compiler produces surprisingly good machine code for tight loops — but the garbage collector is the bottleneck. Every table allocation, every string creation, every closure construction creates GC pressure. Under sustained write workloads, GC pauses dominate latency. We measured p99 write latency of 12ms for a workload of 10,000 writes per second to a 1 million record database, compared to sub-millisecond p99 for RocksDB under the same workload.

Memory efficiency. Lua tables are hash tables, and hash tables are not memory-efficient for storing large numbers of small records. Each table has a header, each key is a full Lua string, and LuaJIT’s memory limit of 2 GB (in the 5.1-compatible mode most applications use) means you run out of address space long before you run out of physical RAM.

Durability guarantees. Implementing a correct WAL in a garbage-collected language is surprisingly tricky. The WAL needs to be flushed to disk before the corresponding data is visible, but the GC can move or collect objects between the write and the flush. We solved this, but the solution added complexity that felt like fighting the language rather than working with it.

Concurrency. Lua has no built-in threading model. LuaJIT has no concurrent GC. Multiple coroutines can share a database, but true parallel access requires multiple Lua states, each with its own copy of the data or a shared-memory mechanism that Lua does not natively support.

The Rust Rewrite: liath-rs

liath-rs keeps the Lua query interface but replaces the storage engine with RocksDB, accessed through Rust. The architecture has three layers:

  1. Lua query interface: The same API as liath v1. Queries are Lua functions, data appears as Lua tables. From the user’s perspective, nothing has changed.

  2. Rust bridge: A Rust library that embeds a Lua runtime (using the mlua crate) and translates between Lua tables and RocksDB’s byte-oriented storage. This layer handles serialisation (Lua tables to MessagePack) and deserialisation (MessagePack to Lua tables).

  3. RocksDB storage: Facebook’s battle-tested LSM-tree storage engine, accessed through Rust’s rocksdb crate. RocksDB handles compaction, bloom filters, compression, and all the other complexities of a production storage engine.

What Changed

Write performance. p99 write latency dropped from 12ms to 0.3ms under the same 10,000 writes/second workload. This is a 40x improvement, and it comes almost entirely from eliminating GC pauses. RocksDB buffers writes in a memtable and flushes to disk asynchronously. There is no garbage collector to pause.

Read performance. For point lookups (fetching a single record by key), liath-rs is approximately 5x faster than liath v1. For full-table scans with Lua query functions, the improvement is approximately 3x. The smaller improvement for scans reflects the fact that the bottleneck shifts from storage access to Lua function evaluation — the query function itself runs at the same speed in both versions.

Memory efficiency. liath-rs uses approximately 60% less memory than liath v1 for the same dataset, because vectors are stored in RocksDB’s compressed SST files rather than in Lua tables. The 2 GB LuaJIT memory limit no longer applies to data storage, only to the Lua runtime and active query state.

Durability. RocksDB’s WAL is well-tested and handles crash recovery correctly. We no longer need to implement our own durability layer.

What Did Not Change

The query model. Queries are still Lua functions. The expressiveness, the ability to use closures and higher-order functions, the tight integration with application logic — all of this is preserved. liath-rs is not a different database from the user’s perspective. It is the same database with a faster engine.

Embeddability. liath-rs still embeds in a Rust process (which can also embed Lua). The deployment model is still “link the library, call the function.” There is no server process.

The learning curve. If you knew how to use liath v1, you know how to use liath-rs. The API is identical.

What the Performance Tells Us

The comparison between liath and liath-rs is instructive beyond the specific numbers.

The storage engine matters more than the query engine for write-heavy workloads. The 40x write performance improvement comes entirely from replacing Lua’s storage layer with RocksDB. The query evaluation code is unchanged. This suggests that for database-like systems, optimising the storage engine yields dramatically higher returns than optimising the query executor.

The query engine matters more than the storage engine for read-heavy analytical workloads. For full-table scans with complex query functions, the improvement is only 3x. The bottleneck is not reading data from disk — it is evaluating the Lua function on each record. This suggests that for analytical workloads, optimising the query execution model (perhaps with JIT compilation of queries, or by pushing predicates into the storage engine) would yield higher returns.

GC pauses dominate tail latency in Lua-based storage engines. The p99 latency improvement is much larger than the mean latency improvement, because GC pauses affect the tail. This is a well-known phenomenon, but it is useful to see it quantified in a controlled comparison.

When to Rewrite vs. When to Optimise

The liath-to-liath-rs transition was a rewrite, not an optimisation. We did not try to make the Lua storage engine faster. We replaced it entirely with a Rust/RocksDB implementation. Was this the right decision?

In this case, yes, for a specific reason: the performance problem was architectural, not implementational. The issue was not that our Lua LSM-tree implementation was poorly written. The issue was that a garbage-collected language is fundamentally unsuitable for a storage engine that needs predictable tail latency. No amount of optimisation within Lua would eliminate GC pauses. The correct response to an architectural mismatch is an architectural change, not tuning.

But we would not generalise this to “always rewrite in Rust.” If the bottleneck had been in the query execution layer — say, the Lua functions were slow because of poor algorithmic complexity — the right response would have been to optimise the queries, not to rewrite the query executor in Rust. Replacing the language only helps when the language is the problem.

Here is our heuristic:

  • If the bottleneck is algorithmic: Optimise the algorithm. The language does not matter.
  • If the bottleneck is a language runtime limitation (GC pauses, memory limits, lack of SIMD): Consider a rewrite of the bottleneck component, not the entire system.
  • If the architecture is wrong: Rewrite the affected subsystem. liath-rs rewrote the storage engine but kept the query interface.
  • If the interface is right but the implementation is slow: Rewrite the implementation behind the same interface. This is what liath-rs did, and it is why the migration was painless.

The fact that liath and liath-rs expose the same Lua query interface was crucial. It meant that switching from one to the other required changing a single line of code — the import statement. Every query function, every application built on liath, worked identically on liath-rs. The rewrite was invisible to users.

Lessons for AI Infrastructure

The liath project taught us several lessons that generalise to AI infrastructure work:

Prototype in the comfortable language. Production-harden the bottleneck. Lua was the right choice for liath v1. It let us iterate on the query model rapidly, without the overhead of a compile-test cycle. Rust was the right choice for the storage engine in liath-rs. The prototype told us what the interface should look like. The rewrite made it fast.

Keep your interfaces stable across rewrites. The reason the liath-to-liath-rs rewrite was successful is that we rewrote the engine, not the interface. If we had changed the query model at the same time as the storage engine, we would have had two variables changing simultaneously and no way to attribute performance differences.

Measure before and after, with the same workload. The controlled comparison between liath and liath-rs is valuable precisely because the query workload is identical. The only variable is the storage engine. This kind of controlled comparison is rare in systems engineering, where rewrites typically change many things at once.

Both liath and liath-rs are open source. We encourage researchers working on programmable databases, embedded scripting for data systems, or AI workflow infrastructure to try them and contribute.