Skip to content

ADR-0005: Baseline-based regression detection

  • Date: 2026-06-24
  • Status: Accepted

Context

Performance work (see ADR-0004) needs a way to prove that an optimization did not regress another generator. Absolute timings are machine-dependent, so a fixed threshold is meaningless.

Decision

Benchmarks support a baseline workflow with defined thresholds:

python -m benchmarks run --iterations 10
python -m benchmarks baseline save --name v0.0.13
python -m benchmarks compare --baseline v0.0.13

Regression thresholds: a >20 % time increase or a >30 % memory increase flags a regression in comparator.py.

Consequences

  • Baselines are versioned snapshots of generator performance, enabling automated detection of regressions before merging.
  • Provides a quantitative gate for optimization work.
  • Baselines are local-only (.benchmarks/baselines/ is gitignored), so the gate is advisory rather than enforced in CI.