Benchmarking
Badgers includes a benchmarking framework to measure performance (time/memory) of all generators. It helps detect regressions when refactoring or optimizing code.
Quick Start
# Run all benchmarks (performance)
python -m benchmarks run
# Filter by module path
python -m benchmarks run --generators tabular_data.outliers
CLI Reference
run — Execute Benchmarks
python -m benchmarks run [--generators FILTER] [--iterations N] [--timeout S]
| Option | Default | Description |
|---|---|---|
--generators |
(all) | Filter by module path prefix, e.g. tabular_data.outliers |
--iterations |
5 |
Number of iterations for performance measurement |
--timeout |
60 |
Timeout in seconds per scenario |
Results are saved as JSON to .benchmarks/results/run_<branch>_<timestamp>.json. This directory is gitignored — results are local-only.
baseline — Manage Baselines
python -m benchmarks baseline save [--name NAME]
python -m benchmarks baseline list
Baselines are snapshots of results used for regression detection. Save a baseline after verifying all checks pass on a known-good commit.
Baselines are stored in .benchmarks/baselines/ (gitignored, local-only).
compare — Detect Regressions
python -m benchmarks compare [--baseline NAME] [--target PATH]
Compares the latest (or specified) results against a baseline and reports regressions:
- Time regression: >20% increase in mean execution time
- Memory regression: >30% increase in peak memory usage
What Gets Measured
Performance Metrics
- Wall-clock time: min, max, mean, median, stddev over N iterations
- Peak memory: measured via
tracemalloc
Adding a New Generator
Create a registration file _<name>.py in benchmarks/generators/<category>/:
from benchmarks.models import GeneratorBenchmark
from benchmarks.registry import register
from benchmarks.scenarios.tabular import SCENARIO_SMALL_BLOBS, SCENARIO_MEDIUM_BLOBS
from badgers.generators.tabular_data.noise import AdditiveNoiseGenerator
register(GeneratorBenchmark(
generator_cls=AdditiveNoiseGenerator,
name="AdditiveNoise",
module_path="tabular_data.noise",
default_params={"distribution": scipy.stats.norm(scale=0.5)},
scenarios=[SCENARIO_SMALL_BLOBS, SCENARIO_MEDIUM_BLOBS],
))
The registry auto-discovers all _*.py files — no other registration needed.
Testing Generators
Generator correctness is tested separately from benchmarks using pytest. See CONTRIBUTING.md for the test structure and conventions, and ADR-0001 for why the two are kept apart.
Architecture
The benchmarks package is built in three layers:
- Registration (
models.py,registry.py,scenarios/,generators/) —Scenariodataclasses define reusable data factories per data type;GeneratorBenchmarkties a generator class to its scenarios; theRegistryauto-discovers_*.pyregistration modules. - Execution (
runner.py,cli.py) —run_performance()measures wall-clock time and peak memory over multiple iterations; the CLI providesrun,baselineandcompare. - Analysis (
comparator.py) —compare_results()diffs two result sets and flags regressions (>20 % time, >30 % memory).
benchmarks/ lives outside the badgers package on purpose: the library must never depend on
it. This is enforced by an import contract — see
ADR-0007.