Chaining generators through pipelines¶
This tutorial shows how to chain different generators using the provided Pipeline class.
Note: this Pipeline is not compatible with sklearn.pipeline.Pipeline.
Goal¶
Compose several defect generators into one pipeline — select, add outliers, corrupt with noise, drop rows — applied in sequence to a single dataset.
How a pipeline works¶
A Pipeline is a small registry of steps. Each step pairs a name with a
generator; calling generate walks the steps in insertion order, feeding
each generator the output of the previous one. Parameters are passed once,
in a params dict keyed by step name. The result is a chain of defects
applied to a single dataset — closer to real dirty data than any single
generator in isolation.
import badgers
badgers.__version__
'0.1.0'
import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import make_blobs
from badgers.core.pipeline import Pipeline
from badgers.generators.tabular_data.imbalance import UndersamplingGenerator
from badgers.core.selectors import ClassSelector
from scipy.stats import norm # for the noise distribution
from badgers.generators.tabular_data.noise import AdditiveNoiseGenerator
Setup random generator¶
from numpy.random import default_rng
seed = 0
rng = default_rng(seed)
The data¶
Fake data from make_blobs (scikit-learn): four well-separated clusters.
Generators act on tabular data only, so the class label travels inside X
as a third column; a selector can then target it by its column position (2).
X, y = make_blobs(centers=4, random_state=0, cluster_std=0.6)
print(X.shape, dict(zip(*np.unique(y, return_counts=True))))
(100, 2) {np.int64(0): np.int64(25), np.int64(1): np.int64(25), np.int64(2): np.int64(25), np.int64(3): np.int64(25)}
Building the pipeline¶
Two steps:
- imbalance —
UndersamplingGeneratorwith aClassSelectorover the label column (position 2). The proportions are the shares of the total row count removed per class: class 0 loses only a few rows (0.05) while classes 2 and 3 are thinned out (0.15 and 0.2); selectors take int column positions, so no name mapping is needed. - noise —
AdditiveNoiseGeneratorwithnorm(scale=0.5): a small Gaussian perturbation added to every remaining cell, after rows are gone.
Order matters: noise after imbalance means noise is added to the surviving rows only. Swapping the steps would perturb rows that are later deleted.
generators = {
'imbalance': UndersamplingGenerator(random_generator=rng),
'noise': AdditiveNoiseGenerator(random_generator=rng)
}
pipeline = Pipeline(generators=generators)
params = {
# selectors take int positions: the class column is 2 of column_stack([X, y])
# proportions = share of the total row count REMOVED per class
'imbalance': {'selector': ClassSelector([0, 1], class_column=2,
proportions={0: 0.05, 1: 0.1, 2: 0.15, 3: 0.2},
rng=rng)},
'noise': {'distribution': norm(scale=0.5)}
}
Xt = pipeline.generate(X=np.column_stack([X, y]).copy(), params=params)
print(X.shape, '->', Xt.shape)
# verify: rows were dropped by the imbalance step
assert Xt.shape[0] < np.column_stack([X, y]).shape[0]
Inspecting the result¶
In the scatter, look for the thinned clusters (classes 2 and 3 nearly gone) and slightly jittered positions from the noise step.
# the imbalance step recorded which of the original rows are selected (True = dropped)
dropped = generators['imbalance'].get_imbalance_labels()
kept_cls = np.asarray(y)[~dropped]
fig, axes = plt.subplots(1, 2, sharex=True, sharey=True, figsize=(8,4))
axes[0].scatter(X[:, 0], X[:, 1], c=y, cmap='tab10', s=15)
axes[1].scatter(Xt[:, 0], Xt[:, 1], c=kept_cls, cmap='tab10', s=15)
axes[0].set_title(f'Original ({len(y)} rows)')
axes[1].set_title(f'Transformed ({len(kept_cls)} rows, labels from get_imbalance_labels)')
plt.tight_layout();
Class counts before and after¶
Class 0 dominates after the imbalance step; classes 2 and 3 keep only a handful of rows.
dropped = generators['imbalance'].get_imbalance_labels()
kept_cls = np.asarray(y)[~dropped]
fig, axes = plt.subplots(1, 2, sharex=True, sharey=True, figsize=(8,4))
classes, nb = np.unique(np.asarray(y), return_counts=True)
axes[0].bar(classes.astype(float), nb, color=[f'C{i}' for i in range(len(classes))])
axes[0].set_xlabel('classes')
axes[0].set_ylabel('number of instances')
classes_t, nb_t = np.unique(kept_cls, return_counts=True)
axes[1].bar(classes_t.astype(float), nb_t, color=[f'C{i}' for i in range(len(classes_t))])
axes[1].set_xlabel('classes')
axes[1].set_ylabel('number of instances')
axes[0].set_title('Original')
axes[1].set_title('Transformed')
plt.tight_layout();
Takeaways¶
- A pipeline stores steps (name, generator, params) and applies them in insertion order.
- Each step's generator must match the pipeline's data type (tabular here).
- Later steps act on the output of earlier steps — order matters.
paramsis keyed by step name — renaming a step means renaming its params entry.