THE COLLECTION · LEARNING FROM SMALL TRIALS

Is the trial working?

A careful way to learn from a small, reversible trial, without moving the goalposts.

Small trials give small, noisy numbers. This tool combines what you believed before the trial with what it showed. It tells you how sure you can be that your trial option is beating today's service by a margin that matters, and what your own pre-set rule says to do next.

What this tool does and doesn't do. It works only from numbers you enter, for an option you're trialling against a comparator. It does not estimate how likely any future described in the Possibility Atlas is, and nobody should read its results that way. It can't fix a weak comparison: if the trial sites or periods were chosen to flatter the option, the answer is biased. It also treats each episode as independent: if the same people or sites account for many episodes, the result will look more certain than it is. Nothing you enter is saved or sent; everything runs in your browser. Page visits are covered by the site's privacy notice.

1. Before any data, write down your rule.

Fill these in before the trial starts, and keep a dated copy. Changing them after you've seen results is the same as moving the goalposts. (This is the idea behind preregistration; Nosek et al., 2018.) The neutral and hopeful views are there to show how sensitive the answer is; they never replace the view your rule names.

Margin that matters (percentage points)How much better must the trial be to count? Expand if the chance is at least (final look)At the final look. Tested under your skeptical view, with unknown outcomes counted against the trial. Stop if the chance is belowTested under your hopeful view, with unknown outcomes counted for the trial. Same at every look. Number of planned looksHow many times you'll check, including the final one. Fix this now. Expand bar before the final lookA stricter bar for early looks, because every extra look gives a lucky run another chance to cross the bar. This is look numberWhich planned look this is. The final look uses the main expand bar.

Starting views for the trial's completion rate

Best guess (%) and how many episodes of evidence it's worth. A view can't be worth more episodes than the comparable evidence behind it. Marketing material isn't comparable evidence. The comparator's rate always starts from the neutral view, so the starting views apply to the trial option only.

Skeptical: no better than today's track recordGuess %Worth Neutral: no viewGuess %Worth Hopeful: what the proposer expectsGuess %Worth

2. Enter what happened.

Withdrawn or cancelled requests stay in "planned". An outcome nobody recorded is "unknown": never delete it and never count it as zero. The example values are synthetic.

Arm	Planned	Completed	Not completed	Unknown
Trial option
Comparator

3. Burden limits.

Do all the burden limits agreed with the people who carry them still hold? Yes No

Burden limits are set with the affected groups or the responsible public body, not by whoever benefits. A breach blocks expansion whatever the numbers say.

Results

Unknown outcomes counted	Starting view	Trial rate (90% range)	Comparator rate (90% range)	Chance the trial beats the comparator by the margin

How to say it:

Avoid saying: "It's [x]% likely to work." "The future is now likely." "The data prove it."

Checking every month

You can run this every month, but each extra look gives chance another opportunity to cross the bar. Fix the number of looks in your dated record and use the stricter bar until the final look; keep the starting views, rule and bands fixed; and report every month, including the hold months. Lowering the bar because you're "so close" makes expanding on noise much more likely. (In a simulation of the worked example, checking monthly at 0.80 made expanding about three to four times as likely when the trial was only at the margin; a 0.95 bar before the final look brought it back to about the single-look level. Those figures are for that example's design only: with more looks, or a comparator closer to your skeptical view, the risk is higher, and a stricter early bar also makes early expansion rare.)

For watching whether a possible future is unfolding. It weighs the evidence you log. It never turns that weight into a probability.

1. Fix your bands before logging anything.

"How many times more expected is this signal if the possibility is unfolding than if it isn't?" Reserve "strong" for direct observation from your own records. These per-signal bands are separate from the verbal bands for the combined weight, below.

WeakModerateStrong

2. Log signals.

Give entries the same source family when they rest on the same underlying source, such as a press release and the articles that quote it. Within a family, only the strongest entry pointing each way counts.

Signal	Direction	Band	Family	Counted?

Bands for the combined weight: under 3.2 not worth more than a bare mention; 3.2–10 substantial; 10–100 strong; over 100 decisive (Jeffreys' scale, as tabulated by Kass and Raftery, 1995).

What a weight can and can't justify

Weight for a possibility can justify keeping its signals under watch and preparing the reversible move named on the atlas card.
Weight against it can justify holding back that move, or reversing one already taken under the reversal condition recorded for it. The card stays in the atlas and the watch continues.
In neither case is the weight, on its own, a trigger for an irreversible commitment, and it never decides whether the future happens.

Method and status

Method: Beta–binomial updating; the chance is computed by numerical integration. On 1 Nov 2026 the page's calculation code was checked against a separate SciPy calculation (350 test cases plus the three worked examples; largest difference in the chance about 0.00001). The companion workbook was checked the same way.

Status: Research method from the Beyond the Curb program, drafted by AI (Claude) for Cognitive Corp and checked with AI-run tests. Not peer reviewed; not validated with practitioners or in field use. Example values are synthetic and illustrative.

Sources: Kass, R.E. & Raftery, A.E. (1995). Bayes factors. Journal of the American Statistical Association 90(430), 773–795. · Haasnoot, M., Kwakkel, J.H., Walker, W.E. & ter Maat, J. (2013). Dynamic adaptive policy pathways: A method for crafting robust decisions for a deeply uncertain world. Global Environmental Change 23(2), 485–498. https://doi.org/10.1016/j.gloenvcha.2012.12.006 · Nosek, B.A., Ebersole, C.R., DeHaven, A.C. & Mellor, D.T. (2018). The preregistration revolution. PNAS 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114

Build BTC-TRIALCHECK-2026-11-01.1