Adding typos to text¶
Goal¶
Corrupt text corpora with typos: swapped letters, case changes, and leetspeak substitutions — on word lists and per document.
In [1]:
Copied!
import badgers
badgers.__version__
import badgers
badgers.__version__
Out[1]:
'0.0.15'
In [2]:
Copied!
from badgers.generators.text.typos import SwapLettersGenerator, LeetSpeakGenerator, SwapCaseGenerator
from badgers.generators.text.typos import SwapLettersGenerator, LeetSpeakGenerator, SwapCaseGenerator
Setup random generator¶
In [3]:
Copied!
from numpy.random import default_rng
seed = 0
rng = default_rng(seed)
from numpy.random import default_rng
seed = 0
rng = default_rng(seed)
Data¶
In [4]:
Copied!
X = "the quick brown fox jumps over the lazy dog".split(' ')
X = "the quick brown fox jumps over the lazy dog".split(' ')
Swapping letter randomly¶
In [5]:
Copied!
# Note: this gen modifies the input list in place -- pass a copy
gen = SwapLettersGenerator(random_generator=rng)
# Note: this gen modifies the input list in place -- pass a copy
gen = SwapLettersGenerator(random_generator=rng)
In [6]:
Copied!
Xt = gen.generate(X.copy(), swap_proba=1)
Xt = gen.generate(X.copy(), swap_proba=1)
In [7]:
Copied!
print('Original:\t'+' '.join(X))
print('Transformed:\t'+' '.join(Xt))
print('Original:\t'+' '.join(X))
print('Transformed:\t'+' '.join(Xt))
Original: the quick brown fox jumps over the lazy dog Transformed: the qucik borwn fox jmups oevr the lzay dog
Leet Speak¶
In [8]:
Copied!
gen = LeetSpeakGenerator(random_generator=rng)
gen = LeetSpeakGenerator(random_generator=rng)
In [9]:
Copied!
Xt = gen.generate(X.copy(), replacement_proba=0.25)
Xt = gen.generate(X.copy(), replacement_proba=0.25)
In [10]:
Copied!
print('Original:\t'+' '.join(X))
print('Transformed:\t'+' '.join(Xt))
print('Original:\t'+' '.join(X))
print('Transformed:\t'+' '.join(Xt))
Original: the quick brown fox jumps over the lazy dog Transformed: the qบi(|c brow^/ fox jumps ov3r the lazy dog
Swap case¶
In [11]:
Copied!
gen = SwapCaseGenerator(random_generator=rng)
gen = SwapCaseGenerator(random_generator=rng)
In [12]:
Copied!
Xt = gen.generate(X.copy(), swapcase_proba=0.25)
Xt = gen.generate(X.copy(), swapcase_proba=0.25)
In [13]:
Copied!
print('Original:\t'+' '.join(X))
print('Transformed:\t'+' '.join(Xt))
print('Original:\t'+' '.join(X))
print('Transformed:\t'+' '.join(Xt))
Original: the quick brown fox jumps over the lazy dog Transformed: thE qUicK bRoWn fOX jUMps oVer The lazy dog
Using pipelines¶
In [14]:
Copied!
from badgers.core.pipeline import Pipeline
from badgers.core.pipeline import Pipeline
In [15]:
Copied!
generators = {'swap_letters': SwapLettersGenerator(random_generator=rng),
'leet_speak': LeetSpeakGenerator(random_generator=rng),
'swap_case': SwapCaseGenerator(random_generator=rng)}
params = {'swap_letters': {'swap_proba':0.5}, 'leet_speak': {'replacement_proba':0.25}, 'swap_case': {'swapcase_proba':0.25}}
pipeline = Pipeline(generators=generators)
Xt = pipeline.generate(X.copy(), params=params)
generators = {'swap_letters': SwapLettersGenerator(random_generator=rng),
'leet_speak': LeetSpeakGenerator(random_generator=rng),
'swap_case': SwapCaseGenerator(random_generator=rng)}
params = {'swap_letters': {'swap_proba':0.5}, 'leet_speak': {'replacement_proba':0.25}, 'swap_case': {'swapcase_proba':0.25}}
pipeline = Pipeline(generators=generators)
Xt = pipeline.generate(X.copy(), params=params)
In [16]:
Copied!
print('Original:\t'+' '.join(X))
print('Transformed:\t'+' '.join(Xt))
print('Original:\t'+' '.join(X))
print('Transformed:\t'+' '.join(Xt))
Original: the quick brown fox jumps over the lazy dog Transformed: ThE ()_uceyek br\X/oN fØX jumPs <>v[-R 7He |_z^y dog
Takeaways¶
- Text generators take a corpus (list of words/documents) and return the same structure, corrupted.
- One rng seeds all generators in the notebook — results are reproducible.
- Generators compose: a pipeline applies several corruptions in sequence.