compere

Overview

compere addresses the economics of human judgement. When the only reliable signal for ordering a set of items is a person comparing two of them, the budget is the constraint: exhaustive comparison grows quadratically and becomes unaffordable past a few dozen items, while random sampling spends much of the budget on pairs whose order was never in doubt.

The formulation is a multi-armed bandit over candidate pairs. compere uses UCB1 to choose which comparison to request next, concentrating effort where the current ordering is least certain and leaving settled regions alone, and Elo to convert the accumulated outcomes into ratings. Selection and rating are separate components, which is the distinction most discussions of this problem collapse.

That separation is worth dwelling on because it changes what a comparison with the alternatives even means. Bradley-Terry, Thurstone and TrueSkill are rating models: they describe how outcomes map to a latent score. UCB1 is a selection policy: it decides which outcome to ask for. compere is explicit that it is UCB1 plus Elo, and not Bradley-Terry, Thurstone or TrueSkill, because a project that is vague about which half it implements cannot be evaluated properly.

The behaviour over a session is exploration followed by concentration. Early comparisons are spread widely because nothing is known. As ratings stabilise, requests move toward boundaries that are still uncertain — the pairs near a cut-off, where a small number of further comparisons could still change the outcome. Items whose position is settled stop being asked about.

It ships as a Python package with a FastAPI service and SQLAlchemy persistence, because in practice this is deployed behind an interface that shows a human two things and records which they preferred, and the useful unit is a service rather than a library call.

Where it loses: Elo assumes transitive preferences with a particular noise model, and human judgement is not always transitive. If your comparisons are noisy in a structured way, or your raters differ systematically, a rating model that represents uncertainty explicitly — TrueSkill in particular — will fit better. compere is also a poor choice when abundant implicit signal already exists; if clicks or conversions are available, asking people is the expensive way to learn something you are already being told.

Practical guidance for evaluating it: run the selection policy against a synthetic set whose true order you know, then measure how the induced ranking converges as the comparison budget grows. Comparing final accuracy at an unlimited budget tells you almost nothing, because every reasonable method converges eventually; the interesting quantity is the ordering error at the budget you can actually afford, which is usually a few hundred comparisons rather than a few thousand. That framing also makes it obvious when adaptive selection is not worth the complexity.