Generating missing values in tabular data¶

This tutorial shows how to generate missing values on pre-existing tabular data and to visualize both the original and the transformed data

In [1]:

import badgers
badgers.__version__

import badgers badgers.__version__

Out[1]:

'0.0.14'

In [2]:

from sklearn.datasets import make_blobs
from badgers.generators.tabular_data.missingness import MissingCompletelyAtRandom, DummyMissingNotAtRandom, DummyMissingAtRandom
import matplotlib.pyplot as plt
import numpy as np

from sklearn.datasets import make_blobs from badgers.generators.tabular_data.missingness import MissingCompletelyAtRandom, DummyMissingNotAtRandom, DummyMissingAtRandom import matplotlib.pyplot as plt import numpy as np

In [3]:

def plot_missing(X, y, Xt):
    """
    Some utility function to generate the plots
    """
    missing_mask = np.isnan(Xt).any(axis=1)
    fig, axes = plt.subplots(1, 2, sharex=True, sharey=True, figsize=(8,4))
    for label in np.unique(y):
        ix = np.where(y == label)
        axes[0].scatter(X[ix,0],X[ix,1], c = f'C{label}', label = f'{label}')
        ix = np.where(y[~missing_mask] == label )
        axes[1].scatter(Xt[~missing_mask][ix,0], Xt[~missing_mask][ix,1], c = f'C{label}', label = f'{label}')
    # plot missing values
    axes[1].scatter(X[missing_mask][:,0],X[missing_mask][:,1],marker='x', color='black', label = 'missing')
    axes[0].set_title('Original')
    axes[1].set_title('Transformed')
    axes[0].set_xlabel('dimension 0', fontsize=10)
    axes[1].set_xlabel('dimension 0', fontsize=10)
    axes[0].set_ylabel('dimension 1', fontsize=10)
    axes[1].set_ylabel('dimension 1', fontsize=10)
    axes[0].legend()
    axes[1].legend()
    return fig, axes

def plot_missing(X, y, Xt): """ Some utility function to generate the plots """ missing_mask = np.isnan(Xt).any(axis=1) fig, axes = plt.subplots(1, 2, sharex=True, sharey=True, figsize=(8,4)) for label in np.unique(y): ix = np.where(y == label) axes[0].scatter(X[ix,0],X[ix,1], c = f'C{label}', label = f'{label}') ix = np.where(y[~missing_mask] == label ) axes[1].scatter(Xt[~missing_mask][ix,0], Xt[~missing_mask][ix,1], c = f'C{label}', label = f'{label}') # plot missing values axes[1].scatter(X[missing_mask][:,0],X[missing_mask][:,1],marker='x', color='black', label = 'missing') axes[0].set_title('Original') axes[1].set_title('Transformed') axes[0].set_xlabel('dimension 0', fontsize=10) axes[1].set_xlabel('dimension 0', fontsize=10) axes[0].set_ylabel('dimension 1', fontsize=10) axes[1].set_ylabel('dimension 1', fontsize=10) axes[0].legend() axes[1].legend() return fig, axes

Setup random generator¶

In [4]:

from numpy.random import default_rng
seed = 0
rng = default_rng(seed)

from numpy.random import default_rng seed = 0 rng = default_rng(seed)

Load and prepare data¶

We first load an existing dataset from sklearn.datasets

In [5]:

X, y = make_blobs(centers=4, random_state=0)

X, y = make_blobs(centers=4, random_state=0)

Generate missing values¶

Missing value mechanisms are usually categorized as missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR).

Missing completely at random (MCAR)¶

The transformer simply replaces values (row, col) with np.nan independently at random.

In [6]:

trf = MissingCompletelyAtRandom(random_generator=rng)
Xt, _ = trf.generate(X.copy(), y, percentage_missing=0.25)

trf = MissingCompletelyAtRandom(random_generator=rng) Xt, _ = trf.generate(X.copy(), y, percentage_missing=0.25)

In [7]:

Xt[:5]

Xt[:5]

Out[7]:

array([[       nan, 3.12315514],
       [       nan, 8.10251088],
       [1.7373078 , 4.42546234],
       [       nan, 4.68194985],
       [2.20656076, 5.50616718]])

In [8]:

fig, axes = plot_missing(X, y, Xt)

fig, axes = plot_missing(X, y, Xt)

Missing at random (MAR)¶

Missing not at random means that the fact that a value is missing correlates with some other features.

The DummyMissingAtRandom transformer replaces a value (row,col) with np.nan depending upon another feature chosen randomly. The probability of missingness depends linearly on the other chosen feature.

In [9]:

trf = DummyMissingAtRandom(random_generator=rng)
Xt, _ = trf.generate(X.copy(), y, percentage_missing=0.25)

trf = DummyMissingAtRandom(random_generator=rng) Xt, _ = trf.generate(X.copy(), y, percentage_missing=0.25)

In [10]:

Xt[:5]

Xt[:5]

Out[10]:

array([[ 0.46546494,  3.12315514],
       [-2.54111268,         nan],
       [ 1.7373078 ,  4.42546234],
       [ 1.1312175 ,  4.68194985],
       [ 2.20656076,  5.50616718]])

In [11]:

fig, axes = plot_missing(X, y, Xt)

fig, axes = plot_missing(X, y, Xt)

Missing not at random (MNAR)¶

Missing not at random means that the value that is missing depends on its own value had it not been missing.

The DummyMissingNotAtRandom simply replaces a value with np.nan with a probability proportional to the original value.

In [12]:

trf = DummyMissingNotAtRandom(random_generator=rng)
Xt, _ = trf.generate(X.copy(), y, percentage_missing=0.25)

trf = DummyMissingNotAtRandom(random_generator=rng) Xt, _ = trf.generate(X.copy(), y, percentage_missing=0.25)

In [13]:

Xt[:5]

Xt[:5]

Out[13]:

array([[0.46546494,        nan],
       [       nan, 8.10251088],
       [       nan, 4.42546234],
       [1.1312175 , 4.68194985],
       [       nan, 5.50616718]])

In [14]:

fig, axes = plot_missing(X, y, Xt)

fig, axes = plot_missing(X, y, Xt)