Skip to content

Benchmarking

Badgers includes a benchmarking framework to measure performance (time/memory) of all generators. It helps detect regressions when refactoring or optimizing code.

Quick Start

# Run all benchmarks (performance)
python -m benchmarks run

# Filter by module path
python -m benchmarks run --generators tabular_data.outliers

CLI Reference

run — Execute Benchmarks

python -m benchmarks run [--generators FILTER] [--iterations N] [--timeout S]
Option Default Description
--generators (all) Filter by module path prefix, e.g. tabular_data.outliers
--iterations 5 Number of iterations for performance measurement
--timeout 60 Timeout in seconds per scenario

Results are saved as JSON to .benchmarks/results/run_<branch>_<timestamp>.json. This directory is gitignored — results are local-only.

baseline — Manage Baselines

python -m benchmarks baseline save [--name NAME]
python -m benchmarks baseline list

Baselines are snapshots of results used for regression detection. Save a baseline after verifying all checks pass on a known-good commit.

Baselines are stored in .benchmarks/baselines/ (gitignored, local-only).

compare — Detect Regressions

python -m benchmarks compare [--baseline NAME] [--target PATH]

Compares the latest (or specified) results against a baseline and reports regressions:

  • Time regression: >20% increase in mean execution time
  • Memory regression: >30% increase in peak memory usage

What Gets Measured

Performance Metrics

  • Wall-clock time: min, max, mean, median, stddev over N iterations
  • Peak memory: measured via tracemalloc

Adding a New Generator

Create a registration file _<name>.py in benchmarks/generators/<category>/:

from benchmarks.models import GeneratorBenchmark
from benchmarks.registry import register
from benchmarks.scenarios.tabular import SCENARIO_SMALL_BLOBS, SCENARIO_MEDIUM_BLOBS
from badgers.generators.tabular_data.noise import AdditiveNoiseGenerator

register(GeneratorBenchmark(
    generator_cls=AdditiveNoiseGenerator,
    name="AdditiveNoise",
    module_path="tabular_data.noise",
    default_params={"distribution": scipy.stats.norm(scale=0.5)},
    scenarios=[SCENARIO_SMALL_BLOBS, SCENARIO_MEDIUM_BLOBS],
))

The registry auto-discovers all _*.py files — no other registration needed.

Testing Generators

Generator correctness is tested separately from benchmarks using pytest. See CONTRIBUTING.md for the test structure and conventions, and ADR-0001 for why the two are kept apart.

Architecture

The benchmarks package is built in three layers:

  1. Registration (models.py, registry.py, scenarios/, generators/) — Scenario dataclasses define reusable data factories per data type; GeneratorBenchmark ties a generator class to its scenarios; the Registry auto-discovers _*.py registration modules.
  2. Execution (runner.py, cli.py) — run_performance() measures wall-clock time and peak memory over multiple iterations; the CLI provides run, baseline and compare.
  3. Analysis (comparator.py) — compare_results() diffs two result sets and flags regressions (>20 % time, >30 % memory).

benchmarks/ lives outside the badgers package on purpose: the library must never depend on it. This is enforced by an import contract — see ADR-0007.