Generating missing data¶
Missingness is the defect where some observed values are simply
not recorded (np.nan in the output). Real pipelines rarely get to
see complete data: sensors drop out, forms are left blank, values
exceed a recording range. The mechanism matters as much as the
amount — the same 20% of missing values behaves very differently
depending on why the values are missing:
| Mechanism | Meaning | In badgers (selector) | Legacy one-parameter |
|---|---|---|---|
| MCAR — missing completely at random | a value goes missing independently of everything else | BernoulliSelector([col], rate=...) |
MissingCompletelyAtRandom |
| MAR — missing at random | missingness depends on another, observed variable (e.g. values of X are high) |
TopQuantileSelector([col], column=..., driver=...) |
DummyMissingAtRandom |
| MNAR — missing not at random | a value's own (unobserved) magnitude drives its missingness | TopQuantileSelector on itself |
DummyMissingNotAtRandom |
The notebook shows both ways of expressing these mechanisms:
- Acting on the observation process — the legacy one-parameter
generators:
percentage_missingdecides how much goes missing, the generator class decides why. They do not use a DGP and nothing is propagated. - Explicit selectors — the same three mechanisms expressed as
selector objects, which decide which cells are selected. Intensity
(
rate/quantile) lives on the selector, not ongenerate().
Goal¶
Hide cells (write NaN) with different mechanisms: completely at random, related to another column (MAR), or related to the hidden value itself (MNAR) — including the causal-graph variant.
import matplotlib.pyplot as plt
import numpy as np
from numpy.random import default_rng
from sklearn.datasets import make_blobs
from badgers.generators.tabular_data.missingness import (
MissingCompletelyAtRandom, DummyMissingAtRandom, DummyMissingNotAtRandom,
)
from badgers.generators.tabular_data.missingness.causal import CausalMissingnessGenerator
import dowhy.gcm as gcm
from badgers.core.dgp.backends import GcmSCMDGP
from badgers.core.selectors import BernoulliSelector, TopQuantileSelector
rng = default_rng(seed=0)
def make_scm(graph):
scm = gcm.StructuralCausalModel(graph)
for node in graph.nodes:
if not list(graph.predecessors(node)):
scm.set_causal_mechanism(node, gcm.EmpiricalDistribution())
else:
scm.set_causal_mechanism(node, gcm.AdditiveNoiseModel(
prediction_model=gcm.ml.create_linear_regressor()))
return scm
C:\Users\siebert\PycharmProjects\badgers\.venv\Lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
from _shared import plot_missing # from docs/tutorials/tabular-data/_shared.py
X, y = make_blobs(centers=4, random_state=0)
# classes are a column of X
Xc = np.column_stack([X, y])
cls = Xc[:, 2].astype(int) # explicit class vector for plotting
print(X.shape)
(100, 2)
Part A — Acting on the observation process (legacy one-parameter, no DGP)¶
These generators take a single percentage_missing value: how much
goes missing. Why it goes missing is hard-coded by the class
(MCAR / MAR / MNAR). No causal model is involved and no propagation
happens.
Missing completely at random (MCAR)¶
MissingCompletelyAtRandom replaces values with np.nan
independently at random — the fact that a value is missing carries no
information.
gen = MissingCompletelyAtRandom(random_generator=rng)
Xt = gen.generate(X.copy(), percentage_missing=0.2)
print(f'rows with a NaN: {np.isnan(Xt).any(axis=1).sum()} / {len(Xt)}')
rows with a NaN: 37 / 100
fig, axes = plot_missing(X, cls, Xt)
Missing at random (MAR)¶
DummyMissingAtRandom makes missingness depend on another,
observed feature: for each column, a reference column is drawn
uniformly at random among the other columns, and the missing
probability grows as the (normalized, reversed) values of that
reference column. Low values of the reference column go missing first. Looking only at observed data one can
partly explain the missingness (it correlates with the reference column), but
it cannot be fully predicted.
gen = DummyMissingAtRandom(random_generator=rng)
Xt = gen.generate(X.copy(), percentage_missing=0.25)
print(f'rows with a NaN: {np.isnan(Xt).any(axis=1).sum()} / {len(Xt)}')
rows with a NaN: 46 / 100
fig, axes = plot_missing(X, cls, Xt)
Missing not at random (MNAR)¶
DummyMissingNotAtRandom makes a value's own magnitude drive its
missingness (unlike the other two generators, percentage_missing
has no default value — pass it explicitly): the larger the original value, the more likely it is
missing. Here the missingness biases the observed data — e.g. peak
values systematically vanish.
gen = DummyMissingNotAtRandom(random_generator=rng)
Xt = gen.generate(X.copy(), percentage_missing=0.25)
print(f'rows with a NaN: {np.isnan(Xt).any(axis=1).sum()} / {len(Xt)}')
rows with a NaN: 46 / 100
fig, axes = plot_missing(X, cls, Xt)
Part B — Explicit selectors on a causal graph¶
CausalMissingnessGenerator expresses the same three mechanisms
with selector objects on a fitted DGP. It writes NaN at the affected
cells of the perturbation nodes and does not propagate: missingness
is a property of the measurement process, not of the data-generating
process (a descendant can still be observed).
import numpy as np
import matplotlib.pyplot as plt
import networkx as nx
import pandas as pd
import dowhy.gcm as gcm
from badgers.core.dgp.backends import GcmSCMDGP
from _shared import generate_linear_data, draw_graph, plot_2panel # from docs/tutorials/tabular-data/_shared.py
Build the graph, generate data, fit the DGP¶
# --- Build the chain X -> Y -> Z ---
chain = nx.DiGraph()
chain.add_edges_from([("X", "Y"), ("Y", "Z")])
draw_graph(chain, title="Chain: X → Y → Z")
plt.show()
# Part B: a fresh RNG stream (independent of Part A)
rng = np.random.default_rng(42)
X, column_mapping = generate_linear_data(
chain, n_samples=100, rng=rng,
coefficients={("X", "Y"): 2.0, ("Y", "Z"): 0.5},
)
print(f"X shape: {X.shape}")
print(f"column_mapping: {column_mapping}")
X shape: (100, 3)
column_mapping: {'X': 0, 'Y': 1, 'Z': 2}
dgp = GcmSCMDGP(make_scm(chain), column_mapping).fit(X.copy())
gen = CausalMissingnessGenerator(random_generator=np.random.default_rng(0), dgp=dgp)
Fitting causal models: 0%| | 0/3 [00:00<?, ?it/s]
Fitting causal mechanism of node X: 0%| | 0/3 [00:00<?, ?it/s]
Fitting causal mechanism of node Y: 0%| | 0/3 [00:00<?, ?it/s]
Fitting causal mechanism of node Z: 0%| | 0/3 [00:00<?, ?it/s]
Fitting causal mechanism of node Z: 100%|██████████| 3/3 [00:00<00:00, 411.26it/s]
MCAR: a value-blind selector¶
BernoulliSelector([1], rate=...) — each row is affected independently
with probability rate.
from badgers.core.selectors import BernoulliSelector, TopQuantileSelector
Xt = gen.generate(
X.copy(),
selector=BernoulliSelector([1], rate=0.5, rng=gen.random_generator),
)
print(f"rows with a NaN: {np.isnan(Xt).any(axis=1).sum()} / {len(Xt)}")
rows with a NaN: 44 / 100
MNAR: values drive their own missingness¶
With the self-mode TopQuantileSelector, the largest values of Y
itself are the ones that go missing (here the top 30%):
Xt_mnar = gen.generate(
X.copy(),
selector=TopQuantileSelector([1], column=1, quantile=0.7),
)
mask = np.isnan(Xt_mnar[:, 1])
print(f"missing rate: {mask.mean():.2f} (target: 0.30)")
print(f"mean of observed Y: {Xt_mnar[~mask, 1].mean():.2f} vs original {X[:, 1].mean():.2f}")
_ = plot_2panel(X, Xt_mnar, x_idx=1, y_idx=2, x_name='Y', y_name='Z')
missing rate: 0.30 (target: 0.30) mean of observed Y: -1.03 vs original -0.11
MAR: a fully-observed reference column¶
With reference-mode TopQuantileSelector, missingness of Y is
driven by the observed value of X (the top half of X values):
Xt_mar = gen.generate(
X.copy(),
selector=TopQuantileSelector([1], column=1, quantile=0.5, driver=0),
)
x = Xt_mar[:, 0]
mask = np.isnan(Xt_mar[:, 1])
print(f"missing rate where X is high: {mask[x >= np.quantile(x, 0.75)].mean():.2f}")
print(f"missing rate where X is low: {mask[x <= np.quantile(x, 0.25)].mean():.2f}")
_ = plot_2panel(X, Xt_mar, x_idx=0, y_idx=1, x_name='X', y_name='Y')
missing rate where X is high: 1.00 missing rate where X is low: 0.00
Summary¶
| Lever | Generator | Uses a DGP? | Propagates? | Mechanism parameter |
|---|---|---|---|---|
| observation process (legacy) | MissingCompletelyAtRandom |
no | no | class = MCAR, percentage_missing = amount |
| observation process (legacy) | DummyMissingAtRandom |
no | no | class = MAR, percentage_missing = amount |
| observation process (legacy) | DummyMissingNotAtRandom |
no | no | class = MNAR, percentage_missing = amount |
| selectors on a causal graph | CausalMissingnessGenerator |
yes (fitted) | no (observation layer) | selector object = mechanism and intensity |
- With the selector version, intensity (
rate/quantile) is configured on the selector, not ongenerate(). - Missingness never cascades: it is a property of the measurement, not of the data-generating process.
- Generators mutate
Xin place — passX.copy()to keep the original.
Takeaways¶
- Missingness only hides cells (NaN); values and shape are untouched.
- The mechanism decides which cells hide: random (MCAR), conditioned on another column (MAR), or on the value itself (MNAR).
- The causal variant is chooser-only: hidden cells do not cascade to descendants.