import badgers
Goal¶
Write a new tabular generator step by step: the GeneratorMixin contract, generate signatures, testing, and benchmark registration.
Creating a (tabular_data) generator: Scenario¶
Label noise is a common data-quality defect: some rows carry the wrong class label, and classifiers trained on such data silently degrade.
The generator here flips a fraction of the class labels to another
class: for a row picked at random with probability flip_rate, the
label is replaced by a different class.
Create the generator class¶
Every generator inherits from GeneratorMixin (so the import comes
first). This enforces implementing a generate(X, **params)
function.
from badgers.core.base import GeneratorMixin
import badgers.core.decorators
class MyGenerator(GeneratorMixin):
def __init__(self):
"""Constructor code goes here"""
pass
def generate(self, X, **params):
"""Generate code goes here"""
pass
Add a random generator¶
That is just best practice so that we can seed the random number generator and reproduce the generation
We add it to the constructor code, give it a type hint (it should be a numpy.random.Generator object) and the type hint should be a numpy.random.Generator object; it is required (no default) -- the user of your generator must pass one
import numpy as np
from numpy.random import default_rng
class MyGenerator(GeneratorMixin):
def __init__(self, random_generator: np.random.Generator):
"""
:param random_generator: A random gen
"""
self.random_generator = random_generator
def generate(self, X, **params):
"""Generate code goes here"""
pass
Add the preprocess_inputs decorator¶
As a convention for tabular data generators, X is expected to be a numpy array.
Note that badgers implements a utility to preprocess inputs such as lists, list of lists, pandas DataFrames and convert them to numpy arrays. So that a user can actually input other data types than just numpy arrays.
This utility comes in the form of a decorator and is called preprocess_inputs.
from badgers.core.decorators.tabular_data import preprocess_inputs
The decorator wraps a generate(self, X, **params) method: it takes
X only; the class labels live in a dedicated column of X.
from badgers.core.decorators.tabular_data import preprocess_inputs
from badgers.core.selectors import AlwaysSelector, BernoulliSelector, get_selected_fields
class MyGenerator(GeneratorMixin):
def __init__(self, random_generator: np.random.Generator):
"""
:param random_generator: A random gen
"""
self.random_generator = random_generator
@preprocess_inputs
def generate(self, X, **params):
"""Generate code goes here"""
pass
Add the code to transform/generate the data.¶
The algorithm goes like this:
- the class labels live INSIDE X, in a dedicated column
(
class_column). - WHICH rows get flipped is decided by a Selector (next section).
- each flagged row gets a new label: a class chosen at random among the classes that are NOT its current label.
X keeps its shape (a copy — generators never mutate their input);
only the label column changes, and generate returns that single
transformed array.
Add a Selector for the flagged rows¶
WHICH rows get flipped is decided by a Selector
(badgers.core.selectors), not by hand-rolled code like
flag = self.random_generator.random(n) < rate inside generate.
A Selector is a callable returning dict[int, spec]: the keys are
column positions, the specs are boolean masks (or probability scores
in [0, 1]) per row. The presets draw with the generator's own rng
(a single rng stream), so the selection is reproducible together
with the rest of the generation:
BernoulliSelector(columns, rate, rng)— one iid Bernoulli flag per row with probabilityrate(completely at random);AlwaysSelector(columns)— every row is selected;TopQuantileSelector/ClassSelector/ ... — value- or class-based rules.
Separating the row-selection rule (the selector) from the defect itself (the flip) is the pattern the shipped generators follow: the same selector presets gate any defect form (missing values, noise, outliers, ...).
# the selector draws the flag: one iid Bernoulli flag per row
X_demo = np.zeros((1000, 2))
selector = BernoulliSelector([1], rate=0.1, rng=default_rng(seed=0))
flag = next(iter(selector(X_demo).values()))
flag.mean(), flag.sum(), flag.dtype
(np.float64(0.089), np.int64(89), dtype('bool'))
from sklearn.preprocessing import StandardScaler
class MyGenerator(GeneratorMixin):
"""
Flip a fraction of class labels to another class.
The class labels are expected to live INSIDE X, in a dedicated
column (`class_column`). Which rows get flipped is decided by a
Selector
(badgers.core.selectors): the selector draws the flag, the
generator only realizes the defect.
"""
def __init__(self, random_generator: np.random.Generator):
"""
:param random_generator: A random gen
"""
self.random_generator = random_generator
@preprocess_inputs
def generate(self, X, *, class_column: int = -1, selector=None):
"""
Flip a fraction of the labels (column `class_column` of X) to a
different class.
:param X: features, with the class labels in column `class_column`
:param class_column: column of X holding the class labels
:param selector: a Selector deciding which rows are selected; defaults to
BernoulliSelector over the label column, bound to the
generator rng (position 1 = the `class` column of the demo
data above)
"""
if selector is None:
# selectors take positions: resolve the (possibly negative)
# class_column into a column position
col = class_column if class_column >= 0 else X.shape[1] + class_column
selector = BernoulliSelector([col], rate=0.1,
rng=self.random_generator)
# the selector draws the flag: one iid Bernoulli per row
flag = next(iter(selector(X).values()))
# each flagged row gets a new label: any class except its own
labels = X[:, class_column]
classes = np.unique(labels)
Xt = X.copy()
flag_idx = np.flatnonzero(flag)
if flag_idx.size:
# vectorized "different class" choice: draw uniformly, shift
# the rows that drew their own class to the next class
k = classes.size
own = np.searchsorted(classes, labels[flag_idx])
choice = self.random_generator.integers(0, k, size=flag_idx.size)
choice = np.where(choice == own, (choice + 1) % k, choice)
Xt[flag_idx, class_column] = classes[choice]
return Xt
Let's try it¶
First, we generate some data using sklearn make_blob utility.
Second, we instantiate our generator and apply the generate function on our data.
Finally, we plot the data using matplotlib.
Create some fake data and plot it¶
from sklearn.datasets import make_blobs
import matplotlib.pyplot as plt
from matplotlib.lines import Line2D
# create a seed that will be used all along
seed = 0
X, y = make_blobs(centers=4, random_state=seed, cluster_std=0.5)
# the labels live INSIDE X as a dedicated column
# (y is only used here to build that column)
Xc = np.column_stack([X, y])
fig, ax = plt.subplots(figsize=(8, 6))
sc = ax.scatter(Xc[:, 0], Xc[:, 1], c=Xc[:, -1], cmap='tab10', s=15)
classes = np.unique(Xc[:, -1])
handles = [
Line2D([], [], marker='o', linestyle='', color=sc.cmap(sc.norm(c)), label=f'class {c}')
for c in classes
]
ax.legend(handles=handles, title='label')
<matplotlib.legend.Legend at 0x1b56037c050>
Instantiate our generator¶
# setting the random gen with our previous seed
random_generator = default_rng(seed=seed)
gen = MyGenerator(random_generator=random_generator)
Transforming the data¶
# generators never mutate their inputs; copies in, copies out
Xt = gen.generate(Xc.copy())
# Some sanity checks:
labels_before = Xc[:, -1]
labels_after = Xt[:, -1]
# feature columns untouched: the defect lives in the label column
assert np.array_equal(Xt[:, :-1], X)
# some labels changed
assert np.any(labels_after != labels_before)
# flipped labels are always a DIFFERENT class than before
changed = labels_after != labels_before
assert np.all(labels_after[changed] != labels_before[changed])
# same shape. Note: same shape is not a library requirement
# (outliers generators append rows); it is what a label flipper means.
Plot the initial and the transformed data¶
fig, axes = plt.subplots(1, 2, figsize=(8,4), sharex=True, sharey=True)
axes[0].scatter(Xc[:, 0], Xc[:, 1], c=Xc[:, -1], cmap='tab10', s=15)
axes[1].scatter(Xt[:, 0], Xt[:, 1], c=Xt[:, -1], cmap='tab10', s=15)
axes[0].set_title('Original')
axes[1].set_title('Transformed')
classes = np.unique(Xc[:, -1])
handles = [
Line2D([], [], marker='o', linestyle='', color=sc.cmap(sc.norm(c)), label=f'class {c}')
for c in classes
]
axes[0].legend(handles=handles, title='label')
axes[1].legend(handles=handles, title='label')
plt.tight_layout()
Let's add some parameters¶
Generators take their parameters as arguments of generate (or at
construction). The selection rule is a Selector: the generator builds
its default BernoulliSelector([1], rate=flip_rate) from the
flip_rate parameter — or a caller-provided selector takes over.
class MyGenerator(GeneratorMixin):
"""
Flip a fraction of class labels to another class
(a realistic name would be ClassFlipGenerator)
"""
def __init__(self, random_generator: np.random.Generator):
"""
:param random_generator: A random gen
"""
self.random_generator = random_generator
@preprocess_inputs
def generate(self, X, *, class_column: int = -1,
flip_rate: float = 0.1, selector=None):
"""
Flip `flip_rate` of the labels to a different class.
:param X: features, with the class labels in column `class_column`
:param class_column: column of X holding the class labels
:param flip_rate: fraction of rows whose label is flipped
(used to build the default BernoulliSelector)
:param selector: a Selector taking over row selection when given
"""
if selector is None:
# selectors take positions: resolve the (possibly negative)
# class_column into a column position
col = class_column if class_column >= 0 else X.shape[1] + class_column
selector = BernoulliSelector([col], rate=flip_rate,
rng=self.random_generator)
flag = next(iter(selector(X).values()))
labels = X[:, class_column]
classes = np.unique(labels)
Xt = X.copy()
flag_idx = np.flatnonzero(flag)
if flag_idx.size:
k = classes.size
own = np.searchsorted(classes, labels[flag_idx])
choice = self.random_generator.integers(0, k, size=flag_idx.size)
choice = np.where(choice == own, (choice + 1) % k, choice)
Xt[flag_idx, class_column] = classes[choice]
return Xt
Instantiate the generator, generate data with different flip_rate values and plot the result¶
Higher flip_rate means more shuffled colors in the transformed
panels — the cluster shapes never move, only the label colors do.
gen = MyGenerator(random_generator=default_rng(seed=0))
labels = Xc[:, -1]
Xt_p01 = gen.generate(Xc.copy(), flip_rate=0.01)
Xt_p10 = gen.generate(Xc.copy(), flip_rate=0.10)
Xt_p50 = gen.generate(Xc.copy(), flip_rate=0.50)
print('flipped:', int((labels != Xt_p01[:, -1]).sum()),
int((labels != Xt_p10[:, -1]).sum()),
int((labels != Xt_p50[:, -1]).sum()))
flipped: 1 11 45
fig, axes = plt.subplots(2, 2, figsize=(8,8), sharex=True, sharey=True)
axes[0,0].scatter(Xc[:, 0], Xc[:, 1], c=labels, cmap='tab10', s=15)
axes[0,1].scatter(Xt_p01[:, 0], Xt_p01[:, 1], c=Xt_p01[:, -1], cmap='tab10', s=15)
axes[1,0].scatter(Xt_p10[:, 0], Xt_p10[:, 1], c=Xt_p10[:, -1], cmap='tab10', s=15)
axes[1,1].scatter(Xt_p50[:, 0], Xt_p50[:, 1], c=Xt_p50[:, -1], cmap='tab10', s=15)
for ax, title in zip(axes.flat, ['flip_rate=0 (original)', 'flip_rate=0.01',
'flip_rate=0.10', 'flip_rate=0.50']):
ax.set_title(title)
plt.tight_layout()
A generator that uses an existing DGP¶
A generator does not have to invent its own mechanism: it can
reuse a fitted DGP object and wrap its sampling or do-
machinery into the generate contract. Build the SCM with the
library of choice (dowhy-gcm or PyMC) and wrap it with an adapter —
GcmSCMDGP (extra pip install badgers[dowhy]) or PyMCSCMDGP
(extra pip install badgers[pymc]); see Using-DGPs.ipynb for the
functionality showcase and the Creating DGP notebooks for
building custom ones.
Here the generator keeps ONE thing in its constructor — the DGP
object (this is structure, exactly like the rng) — and passes
everything else (selector, n_outliers) to
generate.
import networkx as nx
import dowhy.gcm as gcm
from badgers.core.dgp.backends import GcmSCMDGP
# structure first: a causal graph + a fitted DGP object
graph = nx.DiGraph([('A', 'B'), ('B', 'C')])
column_mapping = {'A': 0, 'B': 1, 'C': 2}
rng = default_rng(seed=seed)
n = 500
X_causal = np.zeros((n, 3))
X_causal[:, 0] = rng.normal(size=n)
X_causal[:, 1] = 2.0 * X_causal[:, 0] + rng.normal(scale=0.1, size=n)
X_causal[:, 2] = 0.5 * X_causal[:, 1] + rng.normal(scale=0.1, size=n)
scm = gcm.StructuralCausalModel(graph)
for node in graph.nodes:
if not list(graph.predecessors(node)):
scm.set_causal_mechanism(node, gcm.EmpiricalDistribution())
else:
scm.set_causal_mechanism(node, gcm.AdditiveNoiseModel(
prediction_model=gcm.ml.create_linear_regressor()))
dgp = GcmSCMDGP(scm, column_mapping).fit(X_causal)
print('fitted:', dgp.is_fitted())
C:\Users\siebert\PycharmProjects\badgers\.venv\Lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
Fitting causal models: 0%| | 0/3 [00:00<?, ?it/s]
Fitting causal mechanism of node A: 0%| | 0/3 [00:00<?, ?it/s]
Fitting causal mechanism of node B: 0%| | 0/3 [00:00<?, ?it/s]
Fitting causal mechanism of node C: 0%| | 0/3 [00:00<?, ?it/s]
Fitting causal mechanism of node C: 100%|██████████| 3/3 [00:00<00:00, 389.58it/s]
fitted: True
class DGPGenerator(GeneratorMixin):
"""Push the given rows off their fitted mechanism by
intervening through a DGP object: fresh noise for the
perturbation nodes, descendants recomputed through the
fitted mechanisms."""
def __init__(self, random_generator: np.random.Generator,
dgp=None):
"""
:param random_generator: the rng, as for any generator
:param dgp: a fitted DGP object (structure: given at
construction, like the rng)
"""
self.random_generator = random_generator
self.dgp = dgp
@preprocess_inputs
def generate(self, X, **params):
"""
:param selector: the selector whose columns own the intervention
targets (its selected rows are pushed off mechanism)
:param n_outliers: number of rows to push off mechanism
"""
selector = params['selector']
n_outliers = params['n_outliers']
# the selector owns the targets: select cells, then pick selected rows
selected = get_selected_fields(X, selector, self.random_generator).any(axis=1)
# pick the selected rows to disturb, then let the DGP do the work:
# fresh rows sampled through the fitted mechanisms
row_idx = self.random_generator.choice(np.flatnonzero(selected),
size=n_outliers, replace=False)
Xt = X.copy()
Xt[row_idx] = self.dgp.sample(self.random_generator, n_outliers)
return Xt
# try it: the same DGP object powers a data-quality generator
gen = DGPGenerator(random_generator=default_rng(seed=seed), dgp=dgp)
Xt = gen.generate(X_causal.copy(),
selector=AlwaysSelector([0]), n_outliers=5)
print(X_causal.shape, '->', Xt.shape)
changed_rows = np.flatnonzero(~np.isclose(Xt, X_causal).all(axis=1))
print('perturbed rows:', changed_rows, '(', len(changed_rows), ')')
others = np.setdiff1d(np.arange(n), changed_rows)
print('untouched rows unchanged:', np.allclose(Xt[others], X_causal[others]))
assert len(changed_rows) == 5
assert np.allclose(Xt[others], X_causal[others])
(500, 3) -> (500, 3) perturbed rows: [ 40 92 221 274 436] ( 5 ) untouched rows unchanged: True
# viz: rows pushed off the fitted mechanism at A, propagated to B, C
fig, axes = plt.subplots(1, 3, figsize=(13, 3.8), sharex=True, sharey=True)
for j, (name, ax) in enumerate(zip(['A', 'B', 'C'], axes)):
ax.scatter(X_causal[:, j], np.arange(n), s=8, color='lightgray')
ax.scatter(Xt[changed_rows, j], changed_rows, s=25, color='C3', label='DGP-perturbed rows')
ax.set_title(name)
axes[0].set_ylabel('row index')
axes[0].legend()
plt.tight_layout()
Takeaways¶
- A generator that reuses a DGP stores the DGP object in its
constructor (structure, like the rng) and scenario parameters
in
generate. - Build the SCM with the wrapped library's own API and wrap it
with
GcmSCMDGP/PyMCSCMDGP— there is no default backend. - The do-machinery (severing edges, recomputing descendants) comes from the DGP; the generator only picks rows and passes them through.
Final notes:¶
In badgers all generators are organized first by type of modality (tabular_data, time_series, text, etc.) and by type of data quality problem (noise, outliers, drift, etc.): for instance badgers.generators.tabular_data.noise, badgers.generators.time_series.noise, etc.
By convention, one parent class groups all generators that "belong together".
For instance in badgers.generators.tabular_data.noise there is a NoiseGenerator class that serves as parent for the other generators, for instance GaussianNoiseClassesGenerator and AdditiveNoiseGenerator.
This is sometimes useful to do so. For example for storing intern attributes that all generators of this family will need (that is the case in missing values, or in patterns generation).
Some parameters (like flip_rate in our example) will be more specific and might be called at "generation" time (i.e., once we call generate(X, **params)). The advantage is that one creates only one object and can call the generate function with different parameters.
Which parameters need to be given at construction time and at generation time depends on the use case. In general: keep the minimum in the constructor (a random generator, and objects that define the structure) and pass the scenario parameters to the generate call.
Takeaways¶
- Subclass
GeneratorMixin(or the tabular base class) and implementgenerate. - Instantiate with an rng; never rely on global numpy randomness.
- Register the generator in the benchmark framework to measure its cost.