Skip to content

ADR-0001: Strict separation of tests and benchmarks

  • Date: 2026-06-23
  • Status: Accepted

Context

The benchmarks/ package originally duplicated functional assertions through FunctionalCheck / FunctionalResult, so correctness was asserted in two places with two different sets of expectations. When a generator changed, both had to be updated, and they drifted.

Decision

Unit tests (tests/) and performance benchmarks (benchmarks/) serve distinct purposes and must not overlap.

Concern Unit tests Benchmarks
Purpose Catch logic bugs Measure perf, detect regressions
Data Small, varied (numpy/pandas × 1D/2D) Medium-large, standardized
Data source Per-type conftest.py fixtures benchmarks/scenarios/
Speed Fast (<1 ms each) Measured (timed iterations)
Checks Exact assertions Performance stats only

FunctionalCheck / FunctionalResult were removed.

Consequences

  • tests/ is the single source of truth for correctness.
  • Benchmarks focus exclusively on timing, memory and regression detection.
  • A generator bug can never be "caught" by a benchmark run — that is intentional.