Open Science in AI: Why We Publish Everything
The case for openness in AI research: reproducible methods, falsifiable claims, and saying plainly which of our own claims have not yet been measured.
There was a time when the frontier of AI research was legible. Papers appeared on arXiv with full method sections. Training code was hosted on university servers. Datasets were downloadable. If you wanted to reproduce a result, you could — not easily, not always on the first try, but the information was there.
That time is ending. The most capable models now come from corporate labs that publish capability announcements rather than research papers. Training data is undisclosed. Model weights are proprietary. Evaluation benchmarks are run internally, with no way for outsiders to verify the claims. The most cited “papers” in the field increasingly read like product launches with equations.
This is not a complaint about profit motive. Companies are entitled to their business models. But it is a problem for science. Science requires that claims be falsifiable, that methods be reproducible, and that other researchers can build on your work without asking permission. When the most influential work in a field fails all three criteria, the field has a problem.
Skelf Research exists, in part, as a response to this problem. We publish inspectable code and methodology, and aim to make claims testable by others. Open licensing is the default; where a repository’s licence is unresolved, we state that explicitly. This is not altruism. It is methodology.
Open Source Is Not Enough
Let us be precise about what “open” means, because the term has been diluted to near-meaninglessness. Releasing code on GitHub is a start, but code without context is just code. A repository full of Python scripts does not constitute open science any more than a pile of lab equipment constitutes a published experiment.
Open science requires three things beyond open source:
Runnable experiments. Not just code, but the full pipeline: data, dependencies, configuration, and instructions sufficient for a competent researcher to reproduce the result. “It works on my machine” is not reproducibility.
Documented hypotheses. Every experiment tests a claim. That claim should be stated explicitly before the results are presented. Post-hoc storytelling — running experiments and then constructing a narrative that explains whatever happened — is the most common form of soft fraud in machine learning research. We try to state what we expect and why, then report what actually happened.
Measurable claims. “Our approach is faster” is not a claim. A statement of the form “processes N queries in so many milliseconds on this named hardware, against this named baseline, using this script” is a claim. It can be verified. It can be falsified. It can be compared against future work. We aim for every performance assertion to be either a number with its method and benchmark code beside it, or an explicit statement that it has not yet been measured. The second is less impressive and more useful than a number nobody can reproduce.
These are not novel ideas. They are the basic standards of experimental science, applied to a field that has largely abandoned them in the rush to scale.
How Openness Works Across Our Research
Abstract principles are easy to endorse and hard to practise. Here is how openness plays out concretely across four of our research domains, through four projects that span the range of what we study — including where each falls short of the standard above.
LLM Cognition: promptel
promptel treats prompts as declarative, versionable artefacts. A prompt is written in a small .prompt DSL or in YAML, with typed parameters, constraints such as a token limit, and built-in techniques such as chain-of-thought or few-shot, and the same prompt runs against OpenAI, Anthropic or Groq without changing the specification.
The openness point here is specific: prompts are the most consequential and least inspectable component of modern AI systems. When a company says their AI assistant “follows instructions carefully,” the actual instruction — the system prompt — is proprietary. You cannot inspect it, test it, or compare it to alternatives.
promptel makes prompts concrete objects that anyone can read, modify, and test. A promptel specification is a self-contained description of an LLM interaction. You can run it against a different provider. You can diff it across versions. You can write your own test cases against its declared parameters. The prompt is no longer a trade secret — it is a research artefact.
Safe Computing: memista
memista is a small Rust vector search service that pairs SQLite for metadata with a USearch HNSW index. The research question is whether you need a dedicated vector database or whether lightweight embedded search can serve the same purpose for small corpora.
For that question to be answerable, the benchmarks must be reproducible — and here we have to be honest about where memista stands. It is experimental at v0.1.x, it has been tested below roughly 100,000 vectors, and it has not yet published a recall or latency benchmark. So we make no recall or latency claim for it. What the repository does let you do is run your own: the index parameters are in the source, and the API is three endpoints.
This matters because vector search benchmarks are notoriously sensitive to dataset characteristics, hardware, and measurement methodology. A recall figure that cannot be independently verified is not a finding — it is marketing. We would rather publish no number than that kind.
Edge Intelligence: llamafu
llamafu is a Flutter plugin that runs GGUF language models on mobile hardware through llama.cpp over Dart FFI, with no cloud dependency. The research question is about the practical limits of on-device inference: how much model can you run, at what speed, at what quality, on consumer hardware.
Performance claims about on-device inference are particularly prone to irreproducibility. Results depend on the exact device, the thermal state of the processor, the model quantisation, and the measurement methodology. Latency numbers from a lab benchmark on a cool device differ substantially from real-world performance on a phone that has been in someone’s pocket.
That is exactly why we have not yet published device benchmarks for llamafu. A latency or memory figure is only useful with the device, the model, the quantisation, the thermal state and the measurement script stated alongside it, so that someone with the same phone can rerun it. Until we can publish that whole package, the honest artefact is the code and the plugin API, and anyone quoting llamafu performance should measure it on their own target devices.
Formal Optimisation: savanty
savanty bridges natural language and constraint satisfaction — you describe a discrete constraint problem in English, an LLM translates it into Answer Set Programming, and the Clingo solver either finds a valid answer set or proves that none exists.
This is perhaps the domain where openness matters most, because part of the output has a mathematical guarantee and part does not. The solver’s answer is checkable against the generated program by definition. Whether the generated program faithfully encodes the English is not guaranteed at all. But the more interesting openness question is about the pipeline: how does natural language become a formal specification? Where does the LLM interpretation introduce error? What classes of problems does the pipeline handle reliably, and where does it fail?
We publish the full pipeline, and the repository includes a benchmark set of generated problems — graph colouring, knapsack and others, with deliberately infeasible and tightly constrained instances among them. We have not yet published translation-accuracy results from it. When we do, the failure cases — a misread constraint, a formulation that is satisfiable but does not match the user’s intent — will matter as much as the successes, because they define the boundaries of what the approach can do.
The Cost of Openness
We are not naive about the trade-offs. Publishing everything has real costs.
Competitive disadvantage. Every technique we develop is immediately available to anyone, including well-funded competitors who can execute on it faster than we can. There is no moat in open research. An idea we spent months developing can be replicated in weeks by a team with more resources.
Documentation overhead. Making research reproducible takes significant effort beyond making it work. Cleaning up code for public consumption, writing benchmark documentation, setting up CI pipelines that verify reproducibility — this work is invisible in the output but consumes real engineering hours, and it is the reason several of our projects still carry the label “not yet measured”.
Premature exposure. Publishing work-in-progress means publishing things that are incomplete, sometimes wrong, occasionally embarrassing. There is a permanent public record of every dead end and mistaken assumption. This is uncomfortable. It is also, we believe, how science is supposed to work.
Why It Is Worth It
The costs are real but the benefits compound.
Community contributions. Our open repositories receive bug reports, benchmark comparisons, and occasionally code contributions from researchers we have never met. These contributions are disproportionately valuable because they come from people with different hardware, different use cases, and different assumptions than our own. A bug found by an external researcher using the code in a way we did not anticipate is worth more than ten bugs found by internal testing.
Reproducibility as quality control. The discipline of making experiments reproducible forces us to be more rigorous. When you know someone else will run your benchmark, you check your methodology more carefully. When your claims will be tested, you make more conservative claims. The external accountability makes the internal work better.
Trust. In a field where hype is the default register, verifiable claims are a differentiator. A performance number with its benchmark in the repository carries weight in a way that an unverifiable assertion cannot — and so does a plain statement that something has not been measured yet. Trust is built slowly, through accumulated evidence, and it is the most valuable asset a research organisation can have.
Join the Research
Our code is on GitHub. Every project has an open issue tracker. Every measured claim we make should have a corresponding experiment you can run; where one does not, that is a bug we want reported.
If you find a result you cannot reproduce, file an issue — that is the most valuable contribution you can make. If you run our benchmarks on hardware we have not tested, we want to see the numbers. If you think our methodology is flawed, tell us. We would rather be corrected in public than be wrong in private.
This is not a manifesto about how the world should work. It is a description of how we have chosen to work, and an invitation to hold us to it.