Skip to content

The preprocessing module

This module contains the classes that are used to prepare the data before being passed to the models. There is one Preprocessor per data mode or model component (wide, deeptabular, deepimage and deeptext) with the exception of the deeptext component. In this case, two processors are available: one for the case when no Hugging Face model is used (TextPreprocessor) and another one when a Hugging Face model is used (HFPreprocessor).

WidePreprocessor

Bases: BasePreprocessor

Preprocessor to prepare the wide input dataset

This Preprocessor prepares the data for the wide, linear component. This linear model is implemented via an Embedding layer that is connected to the output neuron. WidePreprocessor numerically encodes all the unique values of all categorical columns wide_cols + crossed_cols. See the Example below.

Parameters:

Name Type Description Default
wide_cols List[str]

List of strings with the name of the columns that will label encoded and passed through the wide component

 required
crossed_cols Optional[List[Tuple[str, str]]]

List of Tuples with the name of the columns that will be 'crossed' and then label encoded. e.g. [('education', 'occupation'), ...]. For binary features, a cross-product transformation is 1 if and only if the constituent features are all 1, and 0 otherwise.

 None

Attributes:

Name Type Description
wide_crossed_cols List

List with the names of all columns that will be label encoded
encoding_dict Dict

Dictionary where the keys are the result of pasting colname + '_' + column value and the values are the corresponding mapped integer.
inverse_encoding_dict Dict

the inverse encoding dictionary
wide_dim int

Dimension of the wide model (i.e. dim of the linear layer)

Examples:

>>> import pandas as pd
>>> from pytorch_widedeep.preprocessing import WidePreprocessor
>>> df = pd.DataFrame({'color': ['r', 'b', 'g'], 'size': ['s', 'n', 'l']})
>>> wide_cols = ['color']
>>> crossed_cols = [('color', 'size')]
>>> wide_preprocessor = WidePreprocessor(wide_cols=wide_cols, crossed_cols=crossed_cols)
>>> X_wide = wide_preprocessor.fit_transform(df)
>>> X_wide
array([[1, 4],
       [2, 5],
       [3, 6]])
>>> wide_preprocessor.encoding_dict
{'color_r': 1, 'color_b': 2, 'color_g': 3, 'color_size_r-s': 4, 'color_size_b-n': 5, 'color_size_g-l': 6}
>>> wide_preprocessor.inverse_transform(X_wide)
  color color_size
0     r        r-s
1     b        b-n
2     g        g-l
Source code in pytorch_widedeep/preprocessing/wide_preprocessor.py 
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
class WidePreprocessor(BasePreprocessor):
    r"""Preprocessor to prepare the wide input dataset

    This Preprocessor prepares the data for the wide, linear component.
    This linear model is implemented via an Embedding layer that is
    connected to the output neuron. `WidePreprocessor` numerically
    encodes all the unique values of all categorical columns `wide_cols +
    crossed_cols`. See the Example below.

    Parameters
    ----------
    wide_cols: List
        List of strings with the name of the columns that will label
        encoded and passed through the `wide` component
    crossed_cols: List, default = None
        List of Tuples with the name of the columns that will be `'crossed'`
        and then label encoded. e.g. _[('education', 'occupation'), ...]_. For
        binary features, a cross-product transformation is 1 if and only if
        the constituent features are all 1, and 0 otherwise.

    Attributes
    ----------
    wide_crossed_cols: List
        List with the names of all columns that will be label encoded
    encoding_dict: Dict
        Dictionary where the keys are the result of pasting `colname + '_' +
        column value` and the values are the corresponding mapped integer.
    inverse_encoding_dict: Dict
        the inverse encoding dictionary
    wide_dim: int
        Dimension of the wide model (i.e. dim of the linear layer)

    Examples
    --------
    >>> import pandas as pd
    >>> from pytorch_widedeep.preprocessing import WidePreprocessor
    >>> df = pd.DataFrame({'color': ['r', 'b', 'g'], 'size': ['s', 'n', 'l']})
    >>> wide_cols = ['color']
    >>> crossed_cols = [('color', 'size')]
    >>> wide_preprocessor = WidePreprocessor(wide_cols=wide_cols, crossed_cols=crossed_cols)
    >>> X_wide = wide_preprocessor.fit_transform(df)
    >>> X_wide
    array([[1, 4],
           [2, 5],
           [3, 6]])
    >>> wide_preprocessor.encoding_dict
    {'color_r': 1, 'color_b': 2, 'color_g': 3, 'color_size_r-s': 4, 'color_size_b-n': 5, 'color_size_g-l': 6}
    >>> wide_preprocessor.inverse_transform(X_wide)
      color color_size
    0     r        r-s
    1     b        b-n
    2     g        g-l
    """

    def __init__(
        self, wide_cols: List[str], crossed_cols: Optional[List[Tuple[str, str]]] = None
    ):
        super(WidePreprocessor, self).__init__()

        self.wide_cols = wide_cols
        self.crossed_cols = crossed_cols

        self.is_fitted = False

    def fit(self, df: pd.DataFrame) -> "WidePreprocessor":
        r"""Fits the Preprocessor and creates required attributes

        Parameters
        ----------
        df: pd.DataFrame
            Input pandas dataframe

        Returns
        -------
        WidePreprocessor
            `WidePreprocessor` fitted object
        """
        df_wide = self._prepare_wide(df)
        self.wide_crossed_cols = df_wide.columns.tolist()
        glob_feature_list = self._make_global_feature_list(
            df_wide[self.wide_crossed_cols]
        )
        # leave 0 for padding/"unseen" categories
        self.encoding_dict = {v: i + 1 for i, v in enumerate(glob_feature_list)}
        self.wide_dim = len(self.encoding_dict)
        self.inverse_encoding_dict = {k: v for v, k in self.encoding_dict.items()}
        self.inverse_encoding_dict[0] = "unseen"

        self.is_fitted = True

        return self

    def transform(self, df: pd.DataFrame) -> np.ndarray:
        r"""
        Parameters
        ----------
        df: pd.DataFrame
            Input pandas dataframe

        Returns
        -------
        np.ndarray
            transformed input dataframe
        """
        check_is_fitted(self, attributes=["encoding_dict"])
        df_wide = self._prepare_wide(df)
        encoded = np.zeros([len(df_wide), len(self.wide_crossed_cols)])
        for col_i, col in enumerate(self.wide_crossed_cols):
            encoded[:, col_i] = df_wide[col].apply(
                lambda x: (
                    self.encoding_dict[col + "_" + str(x)]
                    if col + "_" + str(x) in self.encoding_dict
                    else 0
                )
            )
        return encoded.astype("int64")

    def transform_sample(self, df: pd.DataFrame) -> np.ndarray:
        return self.transform(df)[0]

    def inverse_transform(self, encoded: np.ndarray) -> pd.DataFrame:
        r"""Takes as input the output from the `transform` method and it will
        return the original values.

        Parameters
        ----------
        encoded: np.ndarray
            numpy array with the encoded values that are the output from the
            `transform` method

        Returns
        -------
        pd.DataFrame
            Pandas dataframe with the original values
        """
        decoded = pd.DataFrame(encoded, columns=self.wide_crossed_cols)

        if pd.__version__ >= "2.1.0":
            decoded = decoded.map(lambda x: self.inverse_encoding_dict[x])
        else:
            decoded = decoded.applymap(lambda x: self.inverse_encoding_dict[x])

        for col in decoded.columns:
            rm_str = "".join([col, "_"])
            decoded[col] = decoded[col].apply(lambda x: x.replace(rm_str, ""))
        return decoded

    def fit_transform(self, df: pd.DataFrame) -> np.ndarray:
        """Combines `fit` and `transform`

        Parameters
        ----------
        df: pd.DataFrame
            Input pandas dataframe

        Returns
        -------
        np.ndarray
            transformed input dataframe
        """
        return self.fit(df).transform(df)

    def _make_global_feature_list(self, df: pd.DataFrame) -> List:
        glob_feature_list = []
        for column in df.columns:
            glob_feature_list += self._make_column_feature_list(df[column])
        return glob_feature_list

    def _make_column_feature_list(self, s: pd.Series) -> List:
        return [s.name + "_" + str(x) for x in s.unique()]

    def _cross_cols(self, df: pd.DataFrame):
        df_cc = df.copy()
        crossed_colnames = []
        for cols in self.crossed_cols:
            for c in cols:
                df_cc[c] = df_cc[c].astype("str")
            colname = "_".join(cols)
            df_cc[colname] = df_cc[list(cols)].apply(lambda x: "-".join(x), axis=1)
            crossed_colnames.append(colname)
        return df_cc[crossed_colnames]

    def _prepare_wide(self, df: pd.DataFrame):
        if self.crossed_cols is not None:
            df_cc = self._cross_cols(df)
            return pd.concat([df[self.wide_cols], df_cc], axis=1)
        else:
            return df.copy()[self.wide_cols]

    def __repr__(self) -> str:
        list_of_params: List[str] = ["wide_cols={wide_cols}"]
        if self.crossed_cols is not None:
            list_of_params.append("crossed_cols={crossed_cols}")
        all_params = ", ".join(list_of_params)
        return f"WidePreprocessor({all_params.format(**self.__dict__)})"

fit 

fit(df)

Fits the Preprocessor and creates required attributes

Parameters:

Name Type Description Default
df DataFrame

Input pandas dataframe

 required

Returns:

Type Description
WidePreprocessor

WidePreprocessor fitted object
Source code in pytorch_widedeep/preprocessing/wide_preprocessor.py 
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
def fit(self, df: pd.DataFrame) -> "WidePreprocessor":
    r"""Fits the Preprocessor and creates required attributes

    Parameters
    ----------
    df: pd.DataFrame
        Input pandas dataframe

    Returns
    -------
    WidePreprocessor
        `WidePreprocessor` fitted object
    """
    df_wide = self._prepare_wide(df)
    self.wide_crossed_cols = df_wide.columns.tolist()
    glob_feature_list = self._make_global_feature_list(
        df_wide[self.wide_crossed_cols]
    )
    # leave 0 for padding/"unseen" categories
    self.encoding_dict = {v: i + 1 for i, v in enumerate(glob_feature_list)}
    self.wide_dim = len(self.encoding_dict)
    self.inverse_encoding_dict = {k: v for v, k in self.encoding_dict.items()}
    self.inverse_encoding_dict[0] = "unseen"

    self.is_fitted = True

    return self

transform 

transform(df)

Parameters:

Name Type Description Default
df DataFrame

Input pandas dataframe

 required

Returns:

Type Description
ndarray

transformed input dataframe
Source code in pytorch_widedeep/preprocessing/wide_preprocessor.py 
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
def transform(self, df: pd.DataFrame) -> np.ndarray:
    r"""
    Parameters
    ----------
    df: pd.DataFrame
        Input pandas dataframe

    Returns
    -------
    np.ndarray
        transformed input dataframe
    """
    check_is_fitted(self, attributes=["encoding_dict"])
    df_wide = self._prepare_wide(df)
    encoded = np.zeros([len(df_wide), len(self.wide_crossed_cols)])
    for col_i, col in enumerate(self.wide_crossed_cols):
        encoded[:, col_i] = df_wide[col].apply(
            lambda x: (
                self.encoding_dict[col + "_" + str(x)]
                if col + "_" + str(x) in self.encoding_dict
                else 0
            )
        )
    return encoded.astype("int64")

inverse_transform 

inverse_transform(encoded)

Takes as input the output from the transform method and it will return the original values.

Parameters:

Name Type Description Default
encoded ndarray

numpy array with the encoded values that are the output from the transform method

 required

Returns:

Type Description
DataFrame

Pandas dataframe with the original values
Source code in pytorch_widedeep/preprocessing/wide_preprocessor.py 
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
def inverse_transform(self, encoded: np.ndarray) -> pd.DataFrame:
    r"""Takes as input the output from the `transform` method and it will
    return the original values.

    Parameters
    ----------
    encoded: np.ndarray
        numpy array with the encoded values that are the output from the
        `transform` method

    Returns
    -------
    pd.DataFrame
        Pandas dataframe with the original values
    """
    decoded = pd.DataFrame(encoded, columns=self.wide_crossed_cols)

    if pd.__version__ >= "2.1.0":
        decoded = decoded.map(lambda x: self.inverse_encoding_dict[x])
    else:
        decoded = decoded.applymap(lambda x: self.inverse_encoding_dict[x])

    for col in decoded.columns:
        rm_str = "".join([col, "_"])
        decoded[col] = decoded[col].apply(lambda x: x.replace(rm_str, ""))
    return decoded

fit_transform 

fit_transform(df)

Combines fit and transform

Parameters:

Name Type Description Default
df DataFrame

Input pandas dataframe

 required

Returns:

Type Description
ndarray

transformed input dataframe
Source code in pytorch_widedeep/preprocessing/wide_preprocessor.py 
159
160
161
162
163
164
165
166
167
168
169
170
171
172
def fit_transform(self, df: pd.DataFrame) -> np.ndarray:
    """Combines `fit` and `transform`

    Parameters
    ----------
    df: pd.DataFrame
        Input pandas dataframe

    Returns
    -------
    np.ndarray
        transformed input dataframe
    """
    return self.fit(df).transform(df)

TabPreprocessor

Bases: BasePreprocessor

Preprocessor to prepare the deeptabular component input dataset

Parameters:

Name Type Description Default
cat_embed_cols Optional[Union[List[str], List[Tuple[str, int]]]]

List containing the name of the categorical columns that will be represented by embeddings (e.g. ['education', 'relationship', ...]) or a Tuple with the name and the embedding dimension (e.g.: [ ('education',32), ('relationship',16), ...]).
Param alias: embed_cols

 None
continuous_cols Optional[List[str]]

List with the name of the continuous cols

 None
quantization_setup Optional[Union[int, Dict[str, Union[int, List[float]]]]]

Continuous columns can be turned into categorical via pd.cut. If quantization_setup is an int, all continuous columns will be quantized using this value as the number of bins. Alternatively, a dictionary where the keys are the column names to quantize and the values are the either integers indicating the number of bins or a list of scalars indicating the bin edges can also be used.
Param alias: cols_and_bins

 None
cols_to_scale Optional[Union[List[str], str]]

List with the names of the columns that will be standarised via sklearn's StandardScaler. It can also be the string 'all' in which case all the continuous cols will be scaled.

 None
auto_embed_dim bool

Boolean indicating whether the embedding dimensions will be automatically defined via rule of thumb. See embedding_rule below.

 True
embedding_rule Literal[google, fastai_old, fastai_new]

If auto_embed_dim=True, this is the choice of embedding rule of thumb. Choices are:

  • fastai_new: \(min(600, round(1.6 \times n_{cat}^{0.56}))\)

  • fastai_old: \(min(50, (n_{cat}//{2})+1)\)

  • google: \(min(600, round(n_{cat}^{0.24}))\)

 'fastai_new'
default_embed_dim int

Dimension for the embeddings if the embedding dimension is not provided in the cat_embed_cols parameter and auto_embed_dim is set to False.

 16
with_attention bool

Boolean indicating whether the preprocessed data will be passed to an attention-based model (more precisely a model where all embeddings must have the same dimensions). If True, the param cat_embed_cols must just be a list containing just the categorical column names: e.g. ['education', 'relationship', ...]. This is because they will all be encoded using embeddings of the same dim, which will be specified later when the model is defined.
Param alias: for_transformer, for_matrix_factorization, for_mf

 False
with_cls_token bool

Boolean indicating if a '[CLS]' token will be added to the dataset when using attention-based models. The final hidden state corresponding to this token is used as the aggregated representation for classification and regression tasks. If not, the categorical and/or continuous embeddings will be concatenated before being passed to the final MLP (if present).

 False
shared_embed bool

Boolean indicating if the embeddings will be "shared" when using attention-based models. The idea behind shared_embed is described in the Appendix A in the TabTransformer paper: 'The goal of having column embedding is to enable the model to distinguish the classes in one column from those in the other columns'. In other words, the idea is to let the model learn which column is embedded at the time. See: pytorch_widedeep.models.transformers._layers.SharedEmbeddings.

 False
verbose int
1
scale bool

ℹ️ note: this arg will be removed in upcoming releases. Please use cols_to_scale instead.
Bool indicating whether or not to scale/standarise continuous cols. It is important to emphasize that all the DL models for tabular data in the library also include the possibility of normalising the input continuous features via a BatchNorm or a LayerNorm.
Param alias: scale_cont_cols.

 False
already_standard Optional[List[str]]

ℹ️ note: this arg will be removed in upcoming releases. Please use cols_to_scale instead.
List with the name of the continuous cols that do not need to be scaled/standarised.

 None

Other Parameters:

Name Type Description
**kwargs

pd.cut and StandardScaler related args

Attributes:

Name Type Description
cat_cols List[str]

List containing the names of the categorical columns after processing.
cols_and_bins Optional[Dict[str, Union[int, List[float]]]]

Dictionary containing the quantization setup after processing if quantization is requested. This is derived from the quantization_setup parameter.
quant_args Dict

Dictionary containing arguments passed to pandas.cut() function for quantization.
scale_args Dict

Dictionary containing arguments passed to StandardScaler.
label_encoder LabelEncoder

Instance of LabelEncoder used to encode categorical variables. See pytorch_widedeep.utils.dense_utils.LabelEncoder.
cat_embed_input List

List of tuples with the column name, number of individual values for that column and, if with_attention is set to False, the corresponding embeddings dim, e.g. [('education', 16, 10), ('relationship', 6, 8), ...].
standardize_cols List

List of the columns that will be standardized.
scaler StandardScaler

An instance of sklearn.preprocessing.StandardScaler if standardization is requested.
column_idx Dict

Dictionary where keys are column names and values are column indexes. This is necessary to slice tensors.
quantizer Quantizer

An instance of Quantizer if quantization is requested.
is_fitted bool

Boolean indicating if the preprocessor has been fitted.

Examples:

>>> import pandas as pd
>>> import numpy as np
>>> from pytorch_widedeep.preprocessing import TabPreprocessor
>>> df = pd.DataFrame({'color': ['r', 'b', 'g'], 'size': ['s', 'n', 'l'], 'age': [25, 40, 55]})
>>> cat_embed_cols = [('color',5), ('size',5)]
>>> cont_cols = ['age']
>>> deep_preprocessor = TabPreprocessor(cat_embed_cols=cat_embed_cols, continuous_cols=cont_cols)
>>> X_tab = deep_preprocessor.fit_transform(df)
>>> deep_preprocessor.cat_embed_cols
[('color', 5), ('size', 5)]
>>> deep_preprocessor.column_idx
{'color': 0, 'size': 1, 'age': 2}
>>> cont_df = pd.DataFrame({"col1": np.random.rand(10), "col2": np.random.rand(10) + 1})
>>> cont_cols = ["col1", "col2"]
>>> tab_preprocessor = TabPreprocessor(continuous_cols=cont_cols, quantization_setup=3)
>>> ft_cont_df = tab_preprocessor.fit_transform(cont_df)
>>> # or...
>>> quantization_setup = {'col1': [0., 0.4, 1.], 'col2': [1., 1.4, 2.]}
>>> tab_preprocessor2 = TabPreprocessor(continuous_cols=cont_cols, quantization_setup=quantization_setup)
>>> ft_cont_df2 = tab_preprocessor2.fit_transform(cont_df)
Source code in pytorch_widedeep/preprocessing/tab_preprocessor.py 
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
class TabPreprocessor(BasePreprocessor):
    r"""Preprocessor to prepare the `deeptabular` component input dataset

    Parameters
    ----------
    cat_embed_cols: List, default = None
        List containing the name of the categorical columns that will be
        represented by embeddings (e.g. _['education', 'relationship', ...]_) or
        a Tuple with the name and the embedding dimension (e.g.: _[
        ('education',32), ('relationship',16), ...]_).<br/> Param alias: `embed_cols`
    continuous_cols: List, default = None
        List with the name of the continuous cols
    quantization_setup: int or Dict, default = None
        Continuous columns can be turned into categorical via `pd.cut`. If
        `quantization_setup` is an `int`, all continuous columns will be
        quantized using this value as the number of bins. Alternatively, a
        dictionary where the keys are the column names to quantize and the
        values are the either integers indicating the number of bins or a
        list of scalars indicating the bin edges can also be used.<br/>
        Param alias: `cols_and_bins`
    cols_to_scale: List or str, default = None,
        List with the names of the columns that will be standarised via
        sklearn's `StandardScaler`. It can also be the string `'all'` in
        which case all the continuous cols will be scaled.
    auto_embed_dim: bool, default = True
        Boolean indicating whether the embedding dimensions will be
        automatically defined via rule of thumb. See `embedding_rule`
        below.
    embedding_rule: str, default = 'fastai_new'
        If `auto_embed_dim=True`, this is the choice of embedding rule of
        thumb. Choices are:

        - _fastai_new_: $min(600, round(1.6 \times n_{cat}^{0.56}))$

        - _fastai_old_: $min(50, (n_{cat}//{2})+1)$

        - _google_: $min(600, round(n_{cat}^{0.24}))$
    default_embed_dim: int, default=16
        Dimension for the embeddings if the embedding dimension is not
        provided in the `cat_embed_cols` parameter and `auto_embed_dim` is
        set to `False`.
    with_attention: bool, default = False
        Boolean indicating whether the preprocessed data will be passed to an
        attention-based model (more precisely a model where all embeddings
        must have the same dimensions). If `True`, the param `cat_embed_cols`
        must just be a list containing just the categorical column names:
        e.g.
        _['education', 'relationship', ...]_. This is because they will all be
         encoded using embeddings of the same dim, which will be specified
         later when the model is defined. <br/> Param alias:
         `for_transformer`, `for_matrix_factorization`, `for_mf`
    with_cls_token: bool, default = False
        Boolean indicating if a `'[CLS]'` token will be added to the dataset
        when using attention-based models. The final hidden state
        corresponding to this token is used as the aggregated representation
        for classification and regression tasks. If not, the categorical
        and/or continuous embeddings will be concatenated before being passed
        to the final MLP (if present).
    shared_embed: bool, default = False
        Boolean indicating if the embeddings will be "shared" when using
        attention-based models. The idea behind `shared_embed` is
        described in the Appendix A in the [TabTransformer paper](https://arxiv.org/abs/2012.06678):
        _'The goal of having column embedding is to enable the model to
        distinguish the classes in one column from those in the other
        columns'_. In other words, the idea is to let the model learn which
        column is embedded at the time. See: `pytorch_widedeep.models.transformers._layers.SharedEmbeddings`.
    verbose: int, default = 1
    scale: bool, default = False
        :information_source: **note**: this arg will be removed in upcoming
         releases. Please use `cols_to_scale` instead. <br/> Bool indicating
         whether or not to scale/standarise continuous cols. It is important
         to emphasize that all the DL models for tabular data in the library
         also include the possibility of normalising the input continuous
         features via a `BatchNorm` or a `LayerNorm`. <br/> Param alias:
         `scale_cont_cols`.
    already_standard: List, default = None
        :information_source: **note**: this arg will be removed in upcoming
         releases. Please use `cols_to_scale` instead. <br/> List with the
         name of the continuous cols that do not need to be
         scaled/standarised.

    Other Parameters
    ----------------
    **kwargs: dict
        `pd.cut` and `StandardScaler` related args

    Attributes
    ----------
    cat_cols: List[str]
        List containing the names of the categorical columns after processing.
    cols_and_bins: Optional[Dict[str, Union[int, List[float]]]]
        Dictionary containing the quantization setup after processing if
        quantization is requested. This is derived from the
        quantization_setup parameter.
    quant_args: Dict
        Dictionary containing arguments passed to pandas.cut() function for quantization.
    scale_args: Dict
        Dictionary containing arguments passed to StandardScaler.
    label_encoder: LabelEncoder
        Instance of LabelEncoder used to encode categorical variables.
        See `pytorch_widedeep.utils.dense_utils.LabelEncoder`.
    cat_embed_input: List
        List of tuples with the column name, number of individual values for
        that column and, if `with_attention` is set to `False`, the
        corresponding embeddings dim, e.g. _[('education', 16, 10),
        ('relationship', 6, 8), ...]_.
    standardize_cols: List
        List of the columns that will be standardized.
    scaler: StandardScaler
        An instance of `sklearn.preprocessing.StandardScaler` if standardization
        is requested.
    column_idx: Dict
        Dictionary where keys are column names and values are column indexes.
        This is necessary to slice tensors.
    quantizer: Quantizer
        An instance of `Quantizer` if quantization is requested.
    is_fitted: bool
        Boolean indicating if the preprocessor has been fitted.

    Examples
    --------
    >>> import pandas as pd
    >>> import numpy as np
    >>> from pytorch_widedeep.preprocessing import TabPreprocessor
    >>> df = pd.DataFrame({'color': ['r', 'b', 'g'], 'size': ['s', 'n', 'l'], 'age': [25, 40, 55]})
    >>> cat_embed_cols = [('color',5), ('size',5)]
    >>> cont_cols = ['age']
    >>> deep_preprocessor = TabPreprocessor(cat_embed_cols=cat_embed_cols, continuous_cols=cont_cols)
    >>> X_tab = deep_preprocessor.fit_transform(df)
    >>> deep_preprocessor.cat_embed_cols
    [('color', 5), ('size', 5)]
    >>> deep_preprocessor.column_idx
    {'color': 0, 'size': 1, 'age': 2}
    >>> cont_df = pd.DataFrame({"col1": np.random.rand(10), "col2": np.random.rand(10) + 1})
    >>> cont_cols = ["col1", "col2"]
    >>> tab_preprocessor = TabPreprocessor(continuous_cols=cont_cols, quantization_setup=3)
    >>> ft_cont_df = tab_preprocessor.fit_transform(cont_df)
    >>> # or...
    >>> quantization_setup = {'col1': [0., 0.4, 1.], 'col2': [1., 1.4, 2.]}
    >>> tab_preprocessor2 = TabPreprocessor(continuous_cols=cont_cols, quantization_setup=quantization_setup)
    >>> ft_cont_df2 = tab_preprocessor2.fit_transform(cont_df)
    """

    @alias("with_attention", ["for_transformer", "for_matrix_factorization", "for_mf"])
    @alias("cat_embed_cols", ["embed_cols"])
    @alias("scale", ["scale_cont_cols"])
    @alias("quantization_setup", ["cols_and_bins"])
    def __init__(
        self,
        cat_embed_cols: Optional[Union[List[str], List[Tuple[str, int]]]] = None,
        continuous_cols: Optional[List[str]] = None,
        quantization_setup: Optional[
            Union[int, Dict[str, Union[int, List[float]]]]
        ] = None,
        cols_to_scale: Optional[Union[List[str], str]] = None,
        auto_embed_dim: bool = True,
        embedding_rule: Literal["google", "fastai_old", "fastai_new"] = "fastai_new",
        default_embed_dim: int = 16,
        with_attention: bool = False,
        with_cls_token: bool = False,
        shared_embed: bool = False,
        verbose: int = 1,
        *,
        scale: bool = False,
        already_standard: Optional[List[str]] = None,
        **kwargs,
    ):
        super(TabPreprocessor, self).__init__()

        self.continuous_cols = continuous_cols
        self.quantization_setup = quantization_setup
        self.cols_to_scale = cols_to_scale
        self.scale = scale
        self.already_standard = already_standard
        self.auto_embed_dim = auto_embed_dim
        self.embedding_rule = embedding_rule
        self.default_embed_dim = default_embed_dim
        self.with_attention = with_attention
        self.with_cls_token = with_cls_token
        self.shared_embed = shared_embed
        self.verbose = verbose

        self.quant_args = {
            k: v for k, v in kwargs.items() if k in pd.cut.__code__.co_varnames
        }
        self.scale_args = {
            k: v for k, v in kwargs.items() if k in StandardScaler().get_params()
        }

        self._check_inputs(cat_embed_cols)

        if with_cls_token:
            self.cat_embed_cols = (
                ["cls_token"] + cat_embed_cols  # type: ignore[operator]
                if cat_embed_cols is not None
                else ["cls_token"]
            )
        else:
            self.cat_embed_cols = cat_embed_cols  # type: ignore[assignment]

        self.is_fitted = False

    def fit(self, df: pd.DataFrame) -> BasePreprocessor:  # noqa: C901
        """Fits the Preprocessor and creates required attributes

        Parameters
        ----------
        df: pd.DataFrame
            Input pandas dataframe

        Returns
        -------
        TabPreprocessor
            `TabPreprocessor` fitted object
        """

        df_adj = self._insert_cls_token(df) if self.with_cls_token else df.copy()

        self.column_idx: Dict[str, int] = {}

        # Categorical embeddings logic
        if self.cat_embed_cols is not None or self.quantization_setup is not None:
            self.cat_embed_input: List[Union[Tuple[str, int], Tuple[str, int, int]]] = (
                []
            )

        if self.cat_embed_cols is not None:
            df_cat, cat_embed_dim = self._prepare_categorical(df_adj)

            self.label_encoder = LabelEncoder(
                columns_to_encode=df_cat.columns.tolist(),
                shared_embed=self.shared_embed,
                with_attention=self.with_attention,
            )
            self.label_encoder.fit(df_cat)

            for k, v in self.label_encoder.encoding_dict.items():
                if self.with_attention:
                    self.cat_embed_input.append((k, len(v)))
                else:
                    self.cat_embed_input.append((k, len(v), cat_embed_dim[k]))

            self.column_idx.update({k: v for v, k in enumerate(df_cat.columns)})

        # Continuous columns logic
        if self.continuous_cols is not None:
            df_cont, cont_embed_dim = self._prepare_continuous(df_adj)

            # Standardization logic
            if self.standardize_cols is not None:
                self.scaler = StandardScaler(**self.scale_args).fit(
                    df_cont[self.standardize_cols].values
                )
            elif self.verbose:
                warnings.warn("Continuous columns will not be normalised")

            # Quantization logic
            if self.cols_and_bins is not None:
                # we do not run 'Quantizer.fit' here since in the wild case
                # someone wants standardization and quantization for the same
                # columns, the Quantizer will run on the scaled data
                self.quantizer = Quantizer(self.cols_and_bins, **self.quant_args)

                if self.with_attention:
                    for col, n_cat, _ in cont_embed_dim:
                        self.cat_embed_input.append((col, n_cat))
                else:
                    self.cat_embed_input.extend(cont_embed_dim)

            self.column_idx.update(
                {k: v + len(self.column_idx) for v, k in enumerate(df_cont)}
            )

        self.is_fitted = True

        return self

    def transform(self, df: pd.DataFrame) -> np.ndarray:  # noqa: C901
        """Returns the processed `dataframe` as a np.ndarray

        Parameters
        ----------
        df: pd.DataFrame
            Input pandas dataframe

        Returns
        -------
        np.ndarray
            transformed input dataframe
        """
        check_is_fitted(self, condition=self.is_fitted)

        df_adj = self._insert_cls_token(df) if self.with_cls_token else df.copy()

        if self.cat_embed_cols is not None:
            df_cat = df_adj[self.cat_cols]
            df_cat = self.label_encoder.transform(df_cat)
        if self.continuous_cols is not None:
            df_cont = df_adj[self.continuous_cols]
            # Standardization logic
            if self.standardize_cols:
                df_cont[self.standardize_cols] = self.scaler.transform(
                    df_cont[self.standardize_cols].values
                )
            # Quantization logic
            if self.cols_and_bins is not None:
                # Adjustment so I don't have to override the method
                # in 'ChunkTabPreprocessor'
                if self.quantizer.is_fitted:
                    df_cont = self.quantizer.transform(df_cont)
                else:
                    df_cont = self.quantizer.fit_transform(df_cont)
        try:
            df_deep = pd.concat([df_cat, df_cont], axis=1)
        except NameError:
            try:
                df_deep = df_cat.copy()
            except NameError:
                df_deep = df_cont.copy()

        return df_deep.values

    def transform_sample(self, df: pd.DataFrame) -> np.ndarray:
        return self.transform(df).astype("float")[0]

    def inverse_transform(self, encoded: np.ndarray) -> pd.DataFrame:  # noqa: C901
        r"""Takes as input the output from the `transform` method and it will
        return the original values.

        Parameters
        ----------
        encoded: np.ndarray
            array with the output of the `transform` method

        Returns
        -------
        pd.DataFrame
            Pandas dataframe with the original values
        """
        decoded = pd.DataFrame(encoded, columns=list(self.column_idx.keys()))
        # embeddings back to original category
        if self.cat_embed_cols is not None:
            decoded = self.label_encoder.inverse_transform(decoded)
        if self.continuous_cols is not None:
            # quantized cols to the mid point
            if self.cols_and_bins is not None:
                if self.verbose:
                    print(
                        "Note that quantized cols will be turned into the mid point of "
                        "the corresponding bin"
                    )
                for k, v in self.quantizer.inversed_bins.items():
                    decoded[k] = decoded[k].map(v)
            # continuous_cols back to non-standarised
            try:
                decoded[self.standardize_cols] = self.scaler.inverse_transform(
                    decoded[self.standardize_cols]
                )
            except Exception:  # KeyError:
                pass

        if "cls_token" in decoded.columns:
            decoded.drop("cls_token", axis=1, inplace=True)

        return decoded

    def fit_transform(self, df: pd.DataFrame) -> np.ndarray:
        """Combines `fit` and `transform`

        Parameters
        ----------
        df: pd.DataFrame
            Input pandas dataframe

        Returns
        -------
        np.ndarray
            transformed input dataframe
        """
        return self.fit(df).transform(df)

    def _insert_cls_token(self, df: pd.DataFrame) -> pd.DataFrame:
        df_cls = df.copy()
        df_cls.insert(loc=0, column="cls_token", value="[CLS]")
        return df_cls

    def _prepare_categorical(
        self, df: pd.DataFrame
    ) -> Tuple[pd.DataFrame, Dict[str, int]]:
        if isinstance(self.cat_embed_cols[0], tuple):
            self.cat_cols: List[str] = [emb[0] for emb in self.cat_embed_cols]
            cat_embed_dim: Dict[str, int] = dict(self.cat_embed_cols)  # type: ignore
        else:
            self.cat_cols = self.cat_embed_cols  # type: ignore[assignment]
            if self.auto_embed_dim:
                assert isinstance(self.cat_embed_cols[0], str), (
                    "If 'auto_embed_dim' is 'True' and 'with_attention' is 'False', "
                    "'cat_embed_cols' must be a list of strings with the columns to "
                    "be encoded as embeddings."
                )
                n_cats: Dict[str, int] = {col: df[col].nunique() for col in self.cat_embed_cols}  # type: ignore[misc]
                cat_embed_dim = {
                    col: embed_sz_rule(n_cat, self.embedding_rule)  # type: ignore[misc]
                    for col, n_cat in n_cats.items()
                }
            else:
                cat_embed_dim = {
                    e: self.default_embed_dim for e in self.cat_embed_cols  # type: ignore[misc]
                }  # type: ignore
        return df[self.cat_cols], cat_embed_dim

    def _prepare_continuous(
        self, df: pd.DataFrame
    ) -> Tuple[pd.DataFrame, Optional[List[Tuple[str, int, int]]]]:
        # Standardization logic
        if self.cols_to_scale is not None:
            self.standardize_cols = (
                self.cols_to_scale
                if self.cols_to_scale != "all"
                else self.continuous_cols
            )
        elif self.scale:
            if self.already_standard is not None:
                self.standardize_cols = [
                    c for c in self.continuous_cols if c not in self.already_standard
                ]
            else:
                self.standardize_cols = self.continuous_cols
        else:
            self.standardize_cols = None

        # Quantization logic
        if self.quantization_setup is not None:
            # the quantized columns are then treated as categorical
            quant_cont_embed_input: Optional[List[Tuple[str, int, int]]] = []
            if isinstance(self.quantization_setup, int):
                self.cols_and_bins: Optional[Dict[str, Union[int, List[float]]]] = {}
                for col in self.continuous_cols:
                    self.cols_and_bins[col] = self.quantization_setup
                    quant_cont_embed_input.append(
                        (
                            col,
                            self.quantization_setup,
                            (
                                embed_sz_rule(
                                    self.quantization_setup + 1, self.embedding_rule  # type: ignore[arg-type]
                                )
                                if self.auto_embed_dim
                                else self.default_embed_dim
                            ),
                        )
                    )
            else:
                for col, val in self.quantization_setup.items():
                    if isinstance(val, int):
                        quant_cont_embed_input.append(
                            (
                                col,
                                val,
                                (
                                    embed_sz_rule(
                                        val + 1, self.embedding_rule  # type: ignore[arg-type]
                                    )
                                    if self.auto_embed_dim
                                    else self.default_embed_dim
                                ),
                            )
                        )
                    else:
                        quant_cont_embed_input.append(
                            (
                                col,
                                len(val) - 1,
                                (
                                    embed_sz_rule(len(val), self.embedding_rule)  # type: ignore[arg-type]
                                    if self.auto_embed_dim
                                    else self.default_embed_dim
                                ),
                            )
                        )

                self.cols_and_bins = self.quantization_setup.copy()
        else:
            self.cols_and_bins = None
            quant_cont_embed_input = None

        return df[self.continuous_cols], quant_cont_embed_input

    def _check_inputs(self, cat_embed_cols):  # noqa: C901
        if self.scale or self.already_standard is not None:
            warnings.warn(
                "'scale' and 'already_standard' will be deprecated in the next release. "
                "Please use 'cols_to_scale' instead",
                DeprecationWarning,
                stacklevel=2,
            )

        if self.scale:
            if self.already_standard is not None:
                standardize_cols = [
                    c for c in self.continuous_cols if c not in self.already_standard
                ]
            else:
                standardize_cols = self.continuous_cols
        elif self.cols_to_scale is not None:
            standardize_cols = self.cols_to_scale
        else:
            standardize_cols = None

        if standardize_cols is not None:
            if isinstance(self.quantization_setup, int):
                cols_to_quantize_and_standardize = [
                    c for c in standardize_cols if c in self.continuous_cols
                ]
            elif isinstance(self.quantization_setup, dict):
                cols_to_quantize_and_standardize = [
                    c for c in standardize_cols if c in self.quantization_setup
                ]
            else:
                cols_to_quantize_and_standardize = None
            if cols_to_quantize_and_standardize is not None:
                warnings.warn(
                    f"the following columns: {cols_to_quantize_and_standardize} will be first scaled"
                    " using a StandardScaler and then quantized. Make sure this is what you really want"
                )

        if self.with_cls_token and not self.with_attention:
            warnings.warn(
                "If 'with_cls_token' is set to 'True', 'with_attention' will be automatically "
                "to 'True' if is 'False'",
                UserWarning,
            )
            self.with_attention = True

        if (cat_embed_cols is None) and (self.continuous_cols is None):
            raise ValueError(
                "'cat_embed_cols' and 'continuous_cols' are 'None'. Please, define at least one of the two."
            )

        if (
            cat_embed_cols is not None
            and self.continuous_cols is not None
            and len(np.intersect1d(cat_embed_cols, self.continuous_cols)) > 0
        ):
            overlapping_cols = list(
                np.intersect1d(cat_embed_cols, self.continuous_cols)
            )
            raise ValueError(
                "Currently passing columns as both categorical and continuum is not supported."
                " Please, choose one or the other for the following columns: {}".format(
                    ", ".join(overlapping_cols)
                )
            )
        transformer_error_message = (
            "If with_attention is 'True' cat_embed_cols must be a list "
            " of strings with the columns to be encoded as embeddings."
        )
        if (
            self.with_attention
            and cat_embed_cols is not None
            and isinstance(cat_embed_cols[0], tuple)
        ):
            raise ValueError(transformer_error_message)

    def __repr__(self) -> str:  # noqa: C901
        list_of_params: List[str] = []
        if self.cat_embed_cols is not None:
            list_of_params.append("cat_embed_cols={cat_embed_cols}")
        if self.continuous_cols is not None:
            list_of_params.append("continuous_cols={continuous_cols}")
        if self.quantization_setup is not None:
            list_of_params.append("quantization_setup={quantization_setup}")
        if self.cols_to_scale is not None:
            list_of_params.append("cols_to_scale={cols_to_scale}")
        if not self.auto_embed_dim:
            list_of_params.append("auto_embed_dim={auto_embed_dim}")
        if self.embedding_rule != "fastai_new":
            list_of_params.append("embedding_rule='{embedding_rule}'")
        if self.default_embed_dim != 16:
            list_of_params.append("default_embed_dim={default_embed_dim}")
        if self.with_attention:
            list_of_params.append("with_attention={with_attention}")
        if self.with_cls_token:
            list_of_params.append("with_cls_token={with_cls_token}")
        if self.shared_embed:
            list_of_params.append("shared_embed={shared_embed}")
        if self.verbose != 1:
            list_of_params.append("verbose={verbose}")
        if self.scale:
            list_of_params.append("scale={scale}")
        if self.already_standard is not None:
            list_of_params.append("already_standard={already_standard}")
        if len(self.quant_args) > 0:
            list_of_params.append(
                ", ".join([f"{k}" + "=" + f"{v}" for k, v in self.quant_args.items()])
            )
        if len(self.scale_args) > 0:
            list_of_params.append(
                ", ".join([f"{k}" + "=" + f"{v}" for k, v in self.scale_args.items()])
            )
        all_params = ", ".join(list_of_params)
        return f"TabPreprocessor({all_params.format(**self.__dict__)})"

fit 

fit(df)

Fits the Preprocessor and creates required attributes

Parameters:

Name Type Description Default
df DataFrame

Input pandas dataframe

 required

Returns:

Type Description
TabPreprocessor

TabPreprocessor fitted object
Source code in pytorch_widedeep/preprocessing/tab_preprocessor.py 
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
def fit(self, df: pd.DataFrame) -> BasePreprocessor:  # noqa: C901
    """Fits the Preprocessor and creates required attributes

    Parameters
    ----------
    df: pd.DataFrame
        Input pandas dataframe

    Returns
    -------
    TabPreprocessor
        `TabPreprocessor` fitted object
    """

    df_adj = self._insert_cls_token(df) if self.with_cls_token else df.copy()

    self.column_idx: Dict[str, int] = {}

    # Categorical embeddings logic
    if self.cat_embed_cols is not None or self.quantization_setup is not None:
        self.cat_embed_input: List[Union[Tuple[str, int], Tuple[str, int, int]]] = (
            []
        )

    if self.cat_embed_cols is not None:
        df_cat, cat_embed_dim = self._prepare_categorical(df_adj)

        self.label_encoder = LabelEncoder(
            columns_to_encode=df_cat.columns.tolist(),
            shared_embed=self.shared_embed,
            with_attention=self.with_attention,
        )
        self.label_encoder.fit(df_cat)

        for k, v in self.label_encoder.encoding_dict.items():
            if self.with_attention:
                self.cat_embed_input.append((k, len(v)))
            else:
                self.cat_embed_input.append((k, len(v), cat_embed_dim[k]))

        self.column_idx.update({k: v for v, k in enumerate(df_cat.columns)})

    # Continuous columns logic
    if self.continuous_cols is not None:
        df_cont, cont_embed_dim = self._prepare_continuous(df_adj)

        # Standardization logic
        if self.standardize_cols is not None:
            self.scaler = StandardScaler(**self.scale_args).fit(
                df_cont[self.standardize_cols].values
            )
        elif self.verbose:
            warnings.warn("Continuous columns will not be normalised")

        # Quantization logic
        if self.cols_and_bins is not None:
            # we do not run 'Quantizer.fit' here since in the wild case
            # someone wants standardization and quantization for the same
            # columns, the Quantizer will run on the scaled data
            self.quantizer = Quantizer(self.cols_and_bins, **self.quant_args)

            if self.with_attention:
                for col, n_cat, _ in cont_embed_dim:
                    self.cat_embed_input.append((col, n_cat))
            else:
                self.cat_embed_input.extend(cont_embed_dim)

        self.column_idx.update(
            {k: v + len(self.column_idx) for v, k in enumerate(df_cont)}
        )

    self.is_fitted = True

    return self

transform 

transform(df)

Returns the processed dataframe as a np.ndarray

Parameters:

Name Type Description Default
df DataFrame

Input pandas dataframe

 required

Returns:

Type Description
ndarray

transformed input dataframe
Source code in pytorch_widedeep/preprocessing/tab_preprocessor.py 
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
def transform(self, df: pd.DataFrame) -> np.ndarray:  # noqa: C901
    """Returns the processed `dataframe` as a np.ndarray

    Parameters
    ----------
    df: pd.DataFrame
        Input pandas dataframe

    Returns
    -------
    np.ndarray
        transformed input dataframe
    """
    check_is_fitted(self, condition=self.is_fitted)

    df_adj = self._insert_cls_token(df) if self.with_cls_token else df.copy()

    if self.cat_embed_cols is not None:
        df_cat = df_adj[self.cat_cols]
        df_cat = self.label_encoder.transform(df_cat)
    if self.continuous_cols is not None:
        df_cont = df_adj[self.continuous_cols]
        # Standardization logic
        if self.standardize_cols:
            df_cont[self.standardize_cols] = self.scaler.transform(
                df_cont[self.standardize_cols].values
            )
        # Quantization logic
        if self.cols_and_bins is not None:
            # Adjustment so I don't have to override the method
            # in 'ChunkTabPreprocessor'
            if self.quantizer.is_fitted:
                df_cont = self.quantizer.transform(df_cont)
            else:
                df_cont = self.quantizer.fit_transform(df_cont)
    try:
        df_deep = pd.concat([df_cat, df_cont], axis=1)
    except NameError:
        try:
            df_deep = df_cat.copy()
        except NameError:
            df_deep = df_cont.copy()

    return df_deep.values

inverse_transform 

inverse_transform(encoded)

Takes as input the output from the transform method and it will return the original values.

Parameters:

Name Type Description Default
encoded ndarray

array with the output of the transform method

 required

Returns:

Type Description
DataFrame

Pandas dataframe with the original values
Source code in pytorch_widedeep/preprocessing/tab_preprocessor.py 
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
def inverse_transform(self, encoded: np.ndarray) -> pd.DataFrame:  # noqa: C901
    r"""Takes as input the output from the `transform` method and it will
    return the original values.

    Parameters
    ----------
    encoded: np.ndarray
        array with the output of the `transform` method

    Returns
    -------
    pd.DataFrame
        Pandas dataframe with the original values
    """
    decoded = pd.DataFrame(encoded, columns=list(self.column_idx.keys()))
    # embeddings back to original category
    if self.cat_embed_cols is not None:
        decoded = self.label_encoder.inverse_transform(decoded)
    if self.continuous_cols is not None:
        # quantized cols to the mid point
        if self.cols_and_bins is not None:
            if self.verbose:
                print(
                    "Note that quantized cols will be turned into the mid point of "
                    "the corresponding bin"
                )
            for k, v in self.quantizer.inversed_bins.items():
                decoded[k] = decoded[k].map(v)
        # continuous_cols back to non-standarised
        try:
            decoded[self.standardize_cols] = self.scaler.inverse_transform(
                decoded[self.standardize_cols]
            )
        except Exception:  # KeyError:
            pass

    if "cls_token" in decoded.columns:
        decoded.drop("cls_token", axis=1, inplace=True)

    return decoded

fit_transform 

fit_transform(df)

Combines fit and transform

Parameters:

Name Type Description Default
df DataFrame

Input pandas dataframe

 required

Returns:

Type Description
ndarray

transformed input dataframe
Source code in pytorch_widedeep/preprocessing/tab_preprocessor.py 
476
477
478
479
480
481
482
483
484
485
486
487
488
489
def fit_transform(self, df: pd.DataFrame) -> np.ndarray:
    """Combines `fit` and `transform`

    Parameters
    ----------
    df: pd.DataFrame
        Input pandas dataframe

    Returns
    -------
    np.ndarray
        transformed input dataframe
    """
    return self.fit(df).transform(df)

Quantizer

Helper class to perform the quantization of continuous columns. It is included in this docs for completion, since depending on the value of the parameter 'quantization_setup' of the TabPreprocessor class, that class might have an attribute of type Quantizer. However, this class is designed to always run internally within the TabPreprocessor class.

Parameters:

Name Type Description Default
quantization_setup Dict[str, Union[int, List[float]]]

Dictionary where the keys are the column names to quantize and the values are the either integers indicating the number of bins or a list of scalars indicating the bin edges.

 required
Source code in pytorch_widedeep/preprocessing/tab_preprocessor.py 
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
class Quantizer:
    """Helper class to perform the quantization of continuous columns. It is
    included in this docs for completion, since depending on the value of the
    parameter `'quantization_setup'` of the `TabPreprocessor` class, that
    class might have an attribute of type `Quantizer`. However, this class is
    designed to always run internally within the `TabPreprocessor` class.

    Parameters
    ----------
    quantization_setup: Dict, default = None
        Dictionary where the keys are the column names to quantize and the
        values are the either integers indicating the number of bins or a
        list of scalars indicating the bin edges.
    """

    def __init__(
        self,
        quantization_setup: Dict[str, Union[int, List[float]]],
        **kwargs,
    ):
        self.quantization_setup = quantization_setup
        self.quant_args = kwargs

        self.is_fitted = False

    def fit(self, df: pd.DataFrame) -> "Quantizer":
        self.bins: Dict[str, List[float]] = {}
        for col, bins in self.quantization_setup.items():
            _, self.bins[col] = pd.cut(
                df[col], bins, retbins=True, labels=False, **self.quant_args
            )

        self.inversed_bins: Dict[str, Dict[int, float]] = {}
        for col, bins in self.bins.items():
            self.inversed_bins[col] = {
                k: v
                for k, v in list(
                    zip(
                        range(1, len(bins) + 1),
                        [(a + b) / 2.0 for a, b in zip(bins, bins[1:])],
                    )
                )
            }
            # 0
            self.inversed_bins[col][0] = np.nan

        self.is_fitted = True

        return self

    def transform(self, df: pd.DataFrame) -> pd.DataFrame:
        check_is_fitted(self, condition=self.is_fitted)

        dfc = df.copy()
        for col, bins in self.bins.items():
            dfc[col] = pd.cut(dfc[col], bins, labels=False, **self.quant_args)
            # 0 will be left for numbers outside the bins, i.e. smaller than
            # the smaller boundary or larger than the largest boundary
            dfc[col] = dfc[col] + 1
            dfc[col] = dfc[col].fillna(0)
            dfc[col] = dfc[col].astype(int)

        return dfc

    def fit_transform(self, df: pd.DataFrame) -> pd.DataFrame:
        return self.fit(df).transform(df)

    def __repr__(self) -> str:
        return f"Quantizer(quantization_setup={self.quantization_setup})"

TextPreprocessor

Bases: BasePreprocessor

Preprocessor to prepare the deeptext input dataset

Parameters:

Name Type Description Default
text_col str

column in the input dataframe containing the texts

 required
max_vocab int

Maximum number of tokens in the vocabulary

 30000
min_freq int

Minimum frequency for a token to be part of the vocabulary

 5
maxlen int

Maximum length of the tokenized sequences

 80
pad_first bool

Indicates whether the padding index will be added at the beginning or the end of the sequences

 True
pad_idx int

padding index. Fastai's Tokenizer leaves 0 for the 'unknown' token.

 1
already_processed Optional[bool]

Boolean indicating if the sequence of elements is already processed or prepared. If this is the case, this Preprocessor will simply tokenize and pad the sequence. 

Param aliases: `not_text`. <br/>

This parameter is thought for those cases where the input sequences are already fully processed or are directly not text (e.g. IDs)

 False
word_vectors_path Optional[str]

Path to the pretrained word vectors

 None
n_cpus Optional[int]

number of CPUs to used during the tokenization process

 None
verbose int

Enable verbose output.

 1

Attributes:

Name Type Description
vocab Vocab

an instance of pytorch_widedeep.utils.fastai_transforms.Vocab
embedding_matrix ndarray

Array with the pretrained embeddings

Examples:

>>> import pandas as pd
>>> from pytorch_widedeep.preprocessing import TextPreprocessor
>>> df_train = pd.DataFrame({'text_column': ["life is like a box of chocolates",
... "You never know what you're gonna get"]})
>>> text_preprocessor = TextPreprocessor(text_col='text_column', max_vocab=25, min_freq=1, maxlen=10)
>>> text_preprocessor.fit_transform(df_train)
The vocabulary contains 24 tokens
array([[ 1,  1,  1,  1, 10, 11, 12, 13, 14, 15],
       [ 5,  9, 16, 17, 18,  9, 19, 20, 21, 22]], dtype=int32)
>>> df_te = pd.DataFrame({'text_column': ['you never know what is in the box']})
>>> text_preprocessor.transform(df_te)
array([[ 1,  1,  9, 16, 17, 18, 11,  0,  0, 13]], dtype=int32)
Source code in pytorch_widedeep/preprocessing/text_preprocessor.py 
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
class TextPreprocessor(BasePreprocessor):
    r"""Preprocessor to prepare the ``deeptext`` input dataset

    Parameters
    ----------
    text_col: str
        column in the input dataframe containing the texts
    max_vocab: int, default=30000
        Maximum number of tokens in the vocabulary
    min_freq: int, default=5
        Minimum frequency for a token to be part of the vocabulary
    maxlen: int, default=80
        Maximum length of the tokenized sequences
    pad_first: bool,  default = True
        Indicates whether the padding index will be added at the beginning or the
        end of the sequences
    pad_idx: int, default = 1
        padding index. Fastai's Tokenizer leaves 0 for the 'unknown' token.
    already_processed: bool, Optional, default = False
        Boolean indicating if the sequence of elements is already processed or
        prepared. If this is the case, this Preprocessor will simply tokenize
        and pad the sequence. <br/>

            Param aliases: `not_text`. <br/>

        This parameter is thought for those cases where the input sequences
        are already fully processed or are directly not text (e.g. IDs)
    word_vectors_path: str, Optional
        Path to the pretrained word vectors
    n_cpus: int, Optional, default = None
        number of CPUs to used during the tokenization process
    verbose: int, default 1
        Enable verbose output.

    Attributes
    ----------
    vocab: Vocab
        an instance of `pytorch_widedeep.utils.fastai_transforms.Vocab`
    embedding_matrix: np.ndarray
        Array with the pretrained embeddings

    Examples
    ---------
    >>> import pandas as pd
    >>> from pytorch_widedeep.preprocessing import TextPreprocessor
    >>> df_train = pd.DataFrame({'text_column': ["life is like a box of chocolates",
    ... "You never know what you're gonna get"]})
    >>> text_preprocessor = TextPreprocessor(text_col='text_column', max_vocab=25, min_freq=1, maxlen=10)
    >>> text_preprocessor.fit_transform(df_train)
    The vocabulary contains 24 tokens
    array([[ 1,  1,  1,  1, 10, 11, 12, 13, 14, 15],
           [ 5,  9, 16, 17, 18,  9, 19, 20, 21, 22]], dtype=int32)
    >>> df_te = pd.DataFrame({'text_column': ['you never know what is in the box']})
    >>> text_preprocessor.transform(df_te)
    array([[ 1,  1,  9, 16, 17, 18, 11,  0,  0, 13]], dtype=int32)
    """

    @alias("already_processed", ["not_text"])
    def __init__(
        self,
        text_col: str,
        max_vocab: int = 30000,
        min_freq: int = 5,
        maxlen: int = 80,
        pad_first: bool = True,
        pad_idx: int = 1,
        already_processed: Optional[bool] = False,
        word_vectors_path: Optional[str] = None,
        n_cpus: Optional[int] = None,
        verbose: int = 1,
    ):
        super(TextPreprocessor, self).__init__()

        self.text_col = text_col
        self.max_vocab = max_vocab
        self.min_freq = min_freq
        self.maxlen = maxlen
        self.pad_first = pad_first
        self.pad_idx = pad_idx
        self.already_processed = already_processed
        self.word_vectors_path = word_vectors_path
        self.verbose = verbose
        self.n_cpus = n_cpus if n_cpus is not None else os.cpu_count()

        self.is_fitted = False

    def fit(self, df: pd.DataFrame) -> BasePreprocessor:
        """Builds the vocabulary

        Parameters
        ----------
        df: pd.DataFrame
            Input pandas dataframe

        Returns
        -------
        TextPreprocessor
            `TextPreprocessor` fitted object
        """
        texts = self._read_texts(df)

        tokens = get_texts(texts, self.already_processed, self.n_cpus)

        self.vocab: TVocab = Vocab(
            max_vocab=self.max_vocab,
            min_freq=self.min_freq,
            pad_idx=self.pad_idx,
        ).fit(
            tokens,
        )

        if self.verbose:
            print("The vocabulary contains {} tokens".format(len(self.vocab.stoi)))
        if self.word_vectors_path is not None:
            self.embedding_matrix = build_embeddings_matrix(
                self.vocab, self.word_vectors_path, self.min_freq
            )

        self.is_fitted = True

        return self

    def transform(self, df: pd.DataFrame) -> np.ndarray:
        """Returns the padded, _'numericalised'_ sequences

        Parameters
        ----------
        df: pd.DataFrame
            Input pandas dataframe

        Returns
        -------
        np.ndarray
            Padded, _'numericalised'_ sequences
        """
        check_is_fitted(self, attributes=["vocab"])
        texts = self._read_texts(df)
        tokens = get_texts(texts, self.already_processed, self.n_cpus)
        return self._pad_sequences(tokens)

    def transform_sample(self, text: str) -> np.ndarray:
        """Returns the padded, _'numericalised'_ sequence

        Parameters
        ----------
        text: str
            text to be tokenized and padded

        Returns
        -------
        np.ndarray
            Padded, _'numericalised'_ sequence
        """
        check_is_fitted(self, attributes=["vocab"])
        tokens = get_texts([text], self.already_processed, self.n_cpus)
        return self._pad_sequences(tokens)[0]

    def fit_transform(self, df: pd.DataFrame) -> np.ndarray:
        """Combines `fit` and `transform`

        Parameters
        ----------
        df: pd.DataFrame
            Input pandas dataframe

        Returns
        -------
        np.ndarray
            Padded, _'numericalised'_ sequences
        """
        return self.fit(df).transform(df)

    def inverse_transform(self, padded_seq: np.ndarray) -> pd.DataFrame:
        """Returns the original text plus the added 'special' tokens

        Parameters
        ----------
        padded_seq: np.ndarray
            array with the output of the `transform` method

        Returns
        -------
        pd.DataFrame
            Pandas dataframe with the original text plus the added 'special' tokens
        """
        texts = [self.vocab.inverse_transform(num) for num in padded_seq]
        return pd.DataFrame({self.text_col: texts})

    def _pad_sequences(self, tokens: List[List[str]]) -> np.ndarray:
        sequences = [self.vocab.transform(t) for t in tokens]
        padded_seq = np.array(
            [
                pad_sequences(
                    s,
                    maxlen=self.maxlen,
                    pad_first=self.pad_first,
                    pad_idx=self.pad_idx,
                )
                for s in sequences
            ]
        )
        return padded_seq

    def _read_texts(
        self, df: pd.DataFrame, root_dir: Optional[str] = None
    ) -> List[str]:
        if root_dir is not None:
            if not os.path.exists(root_dir):
                raise ValueError(
                    "root_dir does not exist. Please create it before fitting the preprocessor"
                )
            texts_fnames = df[self.text_col].tolist()
            texts: List[str] = []
            for texts_fname in texts_fnames:
                with open(os.path.join(root_dir, texts_fname), "r") as f:
                    texts.append(f.read().replace("\n", ""))
        else:
            texts = df[self.text_col].tolist()

        return texts

    def _load_vocab(self, vocab: TVocab) -> None:
        self.vocab = vocab

    def __repr__(self) -> str:
        list_of_params: List[str] = ["text_col={text_col}"]
        list_of_params.append("max_vocab={max_vocab}")
        list_of_params.append("min_freq={min_freq}")
        list_of_params.append("maxlen={maxlen}")
        list_of_params.append("pad_first={pad_first}")
        list_of_params.append("pad_idx={pad_idx}")
        list_of_params.append("already_processed={already_processed}")
        if self.word_vectors_path is not None:
            list_of_params.append("word_vectors_path={word_vectors_path}")
        if self.n_cpus is not None:
            list_of_params.append("n_cpus={n_cpus}")
        if self.verbose is not None:
            list_of_params.append("verbose={verbose}")
        all_params = ", ".join(list_of_params)
        return f"TextPreprocessor({all_params.format(**self.__dict__)})"

fit 

fit(df)

Builds the vocabulary

Parameters:

Name Type Description Default
df DataFrame

Input pandas dataframe

 required

Returns:

Type Description
TextPreprocessor

TextPreprocessor fitted object
Source code in pytorch_widedeep/preprocessing/text_preprocessor.py 
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
def fit(self, df: pd.DataFrame) -> BasePreprocessor:
    """Builds the vocabulary

    Parameters
    ----------
    df: pd.DataFrame
        Input pandas dataframe

    Returns
    -------
    TextPreprocessor
        `TextPreprocessor` fitted object
    """
    texts = self._read_texts(df)

    tokens = get_texts(texts, self.already_processed, self.n_cpus)

    self.vocab: TVocab = Vocab(
        max_vocab=self.max_vocab,
        min_freq=self.min_freq,
        pad_idx=self.pad_idx,
    ).fit(
        tokens,
    )

    if self.verbose:
        print("The vocabulary contains {} tokens".format(len(self.vocab.stoi)))
    if self.word_vectors_path is not None:
        self.embedding_matrix = build_embeddings_matrix(
            self.vocab, self.word_vectors_path, self.min_freq
        )

    self.is_fitted = True

    return self

transform 

transform(df)

Returns the padded, 'numericalised' sequences

Parameters:

Name Type Description Default
df DataFrame

Input pandas dataframe

 required

Returns:

Type Description
ndarray

Padded, 'numericalised' sequences
Source code in pytorch_widedeep/preprocessing/text_preprocessor.py 
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
def transform(self, df: pd.DataFrame) -> np.ndarray:
    """Returns the padded, _'numericalised'_ sequences

    Parameters
    ----------
    df: pd.DataFrame
        Input pandas dataframe

    Returns
    -------
    np.ndarray
        Padded, _'numericalised'_ sequences
    """
    check_is_fitted(self, attributes=["vocab"])
    texts = self._read_texts(df)
    tokens = get_texts(texts, self.already_processed, self.n_cpus)
    return self._pad_sequences(tokens)

transform_sample 

transform_sample(text)

Returns the padded, 'numericalised' sequence

Parameters:

Name Type Description Default
text str

text to be tokenized and padded

 required

Returns:

Type Description
ndarray

Padded, 'numericalised' sequence
Source code in pytorch_widedeep/preprocessing/text_preprocessor.py 
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
def transform_sample(self, text: str) -> np.ndarray:
    """Returns the padded, _'numericalised'_ sequence

    Parameters
    ----------
    text: str
        text to be tokenized and padded

    Returns
    -------
    np.ndarray
        Padded, _'numericalised'_ sequence
    """
    check_is_fitted(self, attributes=["vocab"])
    tokens = get_texts([text], self.already_processed, self.n_cpus)
    return self._pad_sequences(tokens)[0]

fit_transform 

fit_transform(df)

Combines fit and transform

Parameters:

Name Type Description Default
df DataFrame

Input pandas dataframe

 required

Returns:

Type Description
ndarray

Padded, 'numericalised' sequences
Source code in pytorch_widedeep/preprocessing/text_preprocessor.py 
179
180
181
182
183
184
185
186
187
188
189
190
191
192
def fit_transform(self, df: pd.DataFrame) -> np.ndarray:
    """Combines `fit` and `transform`

    Parameters
    ----------
    df: pd.DataFrame
        Input pandas dataframe

    Returns
    -------
    np.ndarray
        Padded, _'numericalised'_ sequences
    """
    return self.fit(df).transform(df)

inverse_transform 

inverse_transform(padded_seq)

Returns the original text plus the added 'special' tokens

Parameters:

Name Type Description Default
padded_seq ndarray

array with the output of the transform method

 required

Returns:

Type Description
DataFrame

Pandas dataframe with the original text plus the added 'special' tokens
Source code in pytorch_widedeep/preprocessing/text_preprocessor.py 
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
def inverse_transform(self, padded_seq: np.ndarray) -> pd.DataFrame:
    """Returns the original text plus the added 'special' tokens

    Parameters
    ----------
    padded_seq: np.ndarray
        array with the output of the `transform` method

    Returns
    -------
    pd.DataFrame
        Pandas dataframe with the original text plus the added 'special' tokens
    """
    texts = [self.vocab.inverse_transform(num) for num in padded_seq]
    return pd.DataFrame({self.text_col: texts})

HFPreprocessor

Bases: BasePreprocessor

Text processor to prepare the deeptext input dataset that is a wrapper around HuggingFace's tokenizers.

Following the main phylosophy of the pytorch-widedeep library, this class is designed to be as flexible as possible. Therefore, it is coded so that the user can use it as one would use any HuggingFace tokenizers, or following the API call 'protocol' of the rest of the library.

Parameters:

Name Type Description Default
model_name str

The model name from the transformers library e.g. 'bert-base-uncased'. Currently supported models are those from the families: BERT, RoBERTa, DistilBERT, ALBERT and ELECTRA.

 required
use_fast_tokenizer bool

Whether to use the fast tokenizer from HuggingFace or not

 False
text_col Optional[str]

The column in the input dataframe containing the text data. If this tokenizer is used via the fit and transform methods, this argument is mandatory. If the tokenizer is used via the encode method, this argument is not needed since the input text is passed directly to the encode method.

 None
num_workers Optional[int]

Number of workers to use when preprocessing the text data. If not None, and use_fast_tokenizer is False, the text data will be preprocessed in parallel using the number of workers specified. If use_fast_tokenizer is True, this argument is ignored.

 None
preprocessing_rules Optional[List[Callable[[str], str]]]

A list of functions to be applied to the text data before encoding. This can be useful to clean the text data before encoding. For example, removing html tags, special characters, etc.

 None
tokenizer_params Optional[Dict[str, Any]]

Additional parameters to be passed to the HuggingFace's PreTrainedTokenizer. Parameters to the PreTrainedTokenizer can also be passed via the **kwargs argument

 None
encode_params Optional[Dict[str, Any]]

Additional parameters to be passed to the batch_encode_plus method of the HuggingFace's PreTrainedTokenizer. If the fit and transform methods are used, the encode_params dict parameter is mandatory. If the encode method is used, this parameter is not needed since the input text is passed directly to the encode method.

 None
**kwargs

Additional kwargs to be passed to the model, in particular to the PreTrainedTokenizer class.

 {}

Attributes:

Name Type Description
is_fitted bool

Boolean indicating if the preprocessor has been fitted. This is a HuggingFacea tokenizer, so it is always considered fitted and this attribute is manually set to True internally. This parameter exists for consistency with the rest of the library and because is needed for some functionality in the library.

Examples:

>>> import pandas as pd
>>> from pytorch_widedeep.preprocessing import HFPreprocessor
>>> df = pd.DataFrame({"text": ["this is the first text", "this is the second text"]})
>>> hf_processor_1 = HFPreprocessor(model_name="bert-base-uncased", text_col="text")
>>> X_text_1 = hf_processor_1.fit_transform(df)
>>> texts = ["this is a new text", "this is another text"]
>>> hf_processor_2 = HFPreprocessor(model_name="bert-base-uncased")
>>> X_text_2 = hf_processor_2.encode(texts, max_length=10, padding="max_length", truncation=True)
Source code in pytorch_widedeep/preprocessing/hf_preprocessor.py 
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
class HFPreprocessor(BasePreprocessor):
    """Text processor to prepare the ``deeptext`` input dataset that is a
    wrapper around HuggingFace's tokenizers.

    Following the main phylosophy of the `pytorch-widedeep` library, this
    class is designed to be as flexible as possible. Therefore, it is coded
    so that the user can use it as one would use any HuggingFace tokenizers,
    or following the API call 'protocol' of the rest of the library.

    Parameters
    ----------
    model_name: str
        The model name from the transformers library e.g. _'bert-base-uncased'_.
        Currently supported models are those from the families: BERT, RoBERTa,
        DistilBERT, ALBERT and ELECTRA.
    use_fast_tokenizer: bool, default = False
        Whether to use the fast tokenizer from HuggingFace or not
    text_col: Optional[str], default = None
        The column in the input dataframe containing the text data. If this
        tokenizer is used via the `fit` and `transform` methods, this
        argument is mandatory. If the tokenizer is used via the `encode`
        method, this argument is not needed since the input text is passed
        directly to the `encode` method.
    num_workers: Optional[int], default = None
        Number of workers to use when preprocessing the text data. If not
        None, and `use_fast_tokenizer` is False, the text data will be
        preprocessed in parallel using the number of workers specified. If
        `use_fast_tokenizer` is True, this argument is ignored.
    preprocessing_rules: Optional[List[Callable[[str], str]]], default = None
        A list of functions to be applied to the text data before encoding.
        This can be useful to clean the text data before encoding. For
        example, removing html tags, special characters, etc.
    tokenizer_params: Optional[Dict[str, Any]], default = None
        Additional parameters to be passed to the HuggingFace's
        `PreTrainedTokenizer`. Parameters to the `PreTrainedTokenizer`
        can also be passed via the `**kwargs` argument
    encode_params: Optional[Dict[str, Any]], default = None
        Additional parameters to be passed to the `batch_encode_plus` method
        of the HuggingFace's `PreTrainedTokenizer`. If the `fit` and `transform`
        methods are used, the `encode_params` dict parameter is mandatory. If
        the `encode` method is used, this parameter is not needed since the
        input text is passed directly to the `encode` method.
    **kwargs
        Additional kwargs to be passed to the model, in particular to the
        `PreTrainedTokenizer` class.

    Attributes
    ----------
    is_fitted: bool
        Boolean indicating if the preprocessor has been fitted. This is a
        HuggingFacea tokenizer, so it is always considered fitted and this
        attribute is manually set to True internally. This parameter exists
        for consistency with the rest of the library and because is needed
        for some functionality in the library.

    Examples
    --------
    >>> import pandas as pd
    >>> from pytorch_widedeep.preprocessing import HFPreprocessor
    >>> df = pd.DataFrame({"text": ["this is the first text", "this is the second text"]})
    >>> hf_processor_1 = HFPreprocessor(model_name="bert-base-uncased", text_col="text")
    >>> X_text_1 = hf_processor_1.fit_transform(df)
    >>> texts = ["this is a new text", "this is another text"]
    >>> hf_processor_2 = HFPreprocessor(model_name="bert-base-uncased")
    >>> X_text_2 = hf_processor_2.encode(texts, max_length=10, padding="max_length", truncation=True)
    """

    def __init__(
        self,
        model_name: str,
        *,
        use_fast_tokenizer: bool = False,
        text_col: Optional[str] = None,
        root_dir: Optional[str] = None,
        num_workers: Optional[int] = None,
        preprocessing_rules: Optional[List[Callable[[str], str]]] = None,
        tokenizer_params: Optional[Dict[str, Any]] = None,
        encode_params: Optional[Dict[str, Any]] = None,
        **kwargs,
    ):
        self.model_name = model_name
        self.use_fast_tokenizer = use_fast_tokenizer
        self.text_col = text_col
        self.root_dir = root_dir
        self.num_workers = num_workers
        self.preprocessing_rules = preprocessing_rules
        self.tokenizer_params = tokenizer_params if tokenizer_params is not None else {}
        self.encode_params = encode_params if encode_params is not None else {}

        self._multiprocessing = (
            num_workers is not None and num_workers > 1 and not use_fast_tokenizer
        )

        if kwargs:
            self.tokenizer_params.update(kwargs)

        self.tokenizer = get_tokenizer(
            model_name=self.model_name,
            use_fast_tokenizer=self.use_fast_tokenizer,
            **self.tokenizer_params,
        )

        # A HuggingFace tokenizer is already trained, since we need this
        # attribute elsewhere in the library, we simply set it to True
        self.is_fitted = True

    def encode(self, texts: List[str], **kwargs) -> npt.NDArray[np.int64]:
        """
        Encodes a list of texts. The method is a wrapper around the
        `batch_encode_plus` method of the HuggingFace's tokenizer.

        if 'use_fast_tokenizer' is True, the method will use the `batch_encode_plus`

        Parameters
        ----------
        texts: List[str]
            List of texts to be encoded
        **kwargs
            Additional parameters to be passed to the `batch_encode_plus` method
            of the HuggingFace's tokenizer. If the 'encode_params' dict was passed
            when instantiating the class, that dictionaly will be updated with
            the kwargs passed here.

        Returns
        -------
        np.array
            The encoded texts
        """
        if kwargs:
            self.encode_params.update(kwargs)

        if self.preprocessing_rules:
            if self._multiprocessing:
                texts = self._process_text_parallel(texts)
            else:
                texts = [self._preprocess_text(text) for text in texts]

        if self._multiprocessing:
            input_ids = self._encode_paralell(texts, **self.encode_params)
        else:
            encoded_texts = self.tokenizer.batch_encode_plus(
                texts,
                **self.encode_params,
            )
            input_ids = encoded_texts.get("input_ids")

        self.is_fitted = True

        try:
            output = np.array(input_ids)
        except ValueError:
            warnings.warn(
                "Padding and Truncating parameters were not passed and all input arrays "
                "do not have the same shape. Padding to the longest sequence. "
                "Padding will be done with the index of the pad token for the model",
                UserWarning,
            )
            max_len = max([len(ids) for ids in input_ids])
            output = np.array(
                [
                    np.pad(ids, (self.tokenizer.pad_token_id, max_len - len(ids)))
                    for ids in input_ids
                ]
            )

        return output

    def decode(
        self, input_ids: npt.NDArray[np.int64], skip_special_tokens: bool
    ) -> List[str]:
        """
        Decodes a list of input_ids. The method is a wrapper around the
        `convert_ids_to_tokens` and `convert_tokens_to_string` methods of the
        HuggingFace's tokenizer.

        Parameters
        ----------
        input_ids: npt.NDArray[np.int64]
            The input_ids to be decoded
        skip_special_tokens: bool
            Whether to skip the special tokens or not

        Returns
        -------
        List[str]
            The decoded texts
        """
        texts = [
            self.tokenizer.convert_tokens_to_string(
                self.tokenizer.convert_ids_to_tokens(input_ids[i], skip_special_tokens)
            )
            for i in range(input_ids.shape[0])
        ]
        return texts

    def fit(self, df: pd.DataFrame) -> "HFPreprocessor":
        """
        This method is included for consistency with the rest of the library
        in general and with the `BasePreprocessor` in particular. HuggingFace's
        tokenizers and models are already trained. Therefore, the 'fit' method
        here does nothing other than checking that the 'text_col' parameter is
        not `None`.

        Parameters
        ----------
        df: pd.DataFrame
            The dataframe containing the text data in the column specified by
            the 'text_col' parameter
        """
        if self.text_col is None:
            raise ValueError(
                "'text_col' is None. Please specify the column name containing the text data"
                " if you want to use the 'fit' method"
            )
        return self

    def transform(self, df: pd.DataFrame) -> npt.NDArray[np.int64]:
        """
        Encodes the text data in the input dataframe. This method simply
        calls the `encode` method under the hood. Similar to the `fit` method,
        this method is included for consistency with the rest of the library
        in general and with the `BasePreprocessor` in particular.

        Parameters
        ----------
        df: pd.DataFrame
            The dataframe containing the text data in the column specified by
            the 'text_col' parameter

        Returns
        -------
        np.array
            The encoded texts
        """
        if self.text_col is None:
            raise ValueError(
                "'text_col' is None. Please specify the column name containing the text data"
                " if you want to use the 'fit' method"
            )

        texts = self._read_texts(df, self.root_dir)

        return self.encode(texts)

    def transform_sample(self, text: str) -> npt.NDArray[np.int64]:
        """
        Encodes a single text sample.

        Parameters
        ----------
        text: str
            The text sample to be encoded

        Returns
        -------
        np.array
            The encoded text
        """

        if not self.is_fitted:
            raise ValueError(
                "The `encode` (or `fit`) method must be called before calling `transform_sample`"
            )
        return self.encode([text])[0]

    def fit_transform(self, df: pd.DataFrame) -> npt.NDArray[np.int64]:
        """
        Encodes the text data in the input dataframe.

        Parameters
        ----------
        df: pd.DataFrame
            The dataframe containing the text data in the column specified by
            the 'text_col' parameter

        Returns
        -------
        np.array
            The encoded texts
        """
        return self.fit(df).transform(df)

    def inverse_transform(
        self, input_ids: npt.NDArray[np.int64], skip_special_tokens: bool
    ) -> List[str]:
        """
        Decodes a list of input_ids. The method simply calls the `decode` method
        under the hood.

        Parameters
        ----------
        input_ids: npt.NDArray[np.int64]
            The input_ids to be decoded
        skip_special_tokens: bool
            Whether to skip the special tokens or not

        Returns
        -------
        List[str]
            The decoded texts
        """
        return self.decode(input_ids, skip_special_tokens)

    def _process_text_parallel(self, texts: List[str]) -> List[str]:
        num_processes = (
            self.num_workers if self.num_workers is not None else os.cpu_count()
        )
        with ProcessPoolExecutor(max_workers=num_processes) as executor:
            processed_texts = list(executor.map(self._preprocess_text, texts))
        return processed_texts

    def _preprocess_text(self, text: str) -> str:
        for rule in self.preprocessing_rules:
            text = rule(text)
        return text

    def _encode_paralell(self, texts: List[str], **kwargs) -> List[List[int]]:
        num_processes = (
            self.num_workers if self.num_workers is not None else os.cpu_count()
        )
        ptokenizer = partial(self.tokenizer.encode, **kwargs)
        with ProcessPoolExecutor(max_workers=num_processes) as executor:
            encoded_texts = list(executor.map(ptokenizer, texts))
        return encoded_texts

    def _read_texts(
        self, df: pd.DataFrame, root_dir: Optional[str] = None
    ) -> List[str]:
        if root_dir is not None:
            if not os.path.exists(root_dir):
                raise ValueError(
                    "root_dir does not exist. Please create it before fitting the preprocessor"
                )
            texts_fnames = df[self.text_col].tolist()
            texts: List[str] = []
            for texts_fname in texts_fnames:
                with open(os.path.join(root_dir, texts_fname), "r") as f:
                    texts.append(f.read().replace("\n", ""))
        else:
            texts = df[self.text_col].tolist()

        return texts

    def __repr__(self):
        return (
            f"HFPreprocessor(text_col={self.text_col}, model_name={self.model_name}, "
            f"use_fast_tokenizer={self.use_fast_tokenizer}, num_workers={self.num_workers}, "
            f"preprocessing_rules={self.preprocessing_rules}, tokenizer_params={self.tokenizer_params}, "
            f"encode_params={self.encode_params})"
        )

encode 

encode(texts, **kwargs)

Encodes a list of texts. The method is a wrapper around the batch_encode_plus method of the HuggingFace's tokenizer.

if 'use_fast_tokenizer' is True, the method will use the batch_encode_plus

Parameters:

Name Type Description Default
texts List[str]

List of texts to be encoded

 required
**kwargs

Additional parameters to be passed to the batch_encode_plus method of the HuggingFace's tokenizer. If the 'encode_params' dict was passed when instantiating the class, that dictionaly will be updated with the kwargs passed here.

 {}

Returns:

Type Description
array

The encoded texts
Source code in pytorch_widedeep/preprocessing/hf_preprocessor.py 
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
def encode(self, texts: List[str], **kwargs) -> npt.NDArray[np.int64]:
    """
    Encodes a list of texts. The method is a wrapper around the
    `batch_encode_plus` method of the HuggingFace's tokenizer.

    if 'use_fast_tokenizer' is True, the method will use the `batch_encode_plus`

    Parameters
    ----------
    texts: List[str]
        List of texts to be encoded
    **kwargs
        Additional parameters to be passed to the `batch_encode_plus` method
        of the HuggingFace's tokenizer. If the 'encode_params' dict was passed
        when instantiating the class, that dictionaly will be updated with
        the kwargs passed here.

    Returns
    -------
    np.array
        The encoded texts
    """
    if kwargs:
        self.encode_params.update(kwargs)

    if self.preprocessing_rules:
        if self._multiprocessing:
            texts = self._process_text_parallel(texts)
        else:
            texts = [self._preprocess_text(text) for text in texts]

    if self._multiprocessing:
        input_ids = self._encode_paralell(texts, **self.encode_params)
    else:
        encoded_texts = self.tokenizer.batch_encode_plus(
            texts,
            **self.encode_params,
        )
        input_ids = encoded_texts.get("input_ids")

    self.is_fitted = True

    try:
        output = np.array(input_ids)
    except ValueError:
        warnings.warn(
            "Padding and Truncating parameters were not passed and all input arrays "
            "do not have the same shape. Padding to the longest sequence. "
            "Padding will be done with the index of the pad token for the model",
            UserWarning,
        )
        max_len = max([len(ids) for ids in input_ids])
        output = np.array(
            [
                np.pad(ids, (self.tokenizer.pad_token_id, max_len - len(ids)))
                for ids in input_ids
            ]
        )

    return output

decode 

decode(input_ids, skip_special_tokens)

Decodes a list of input_ids. The method is a wrapper around the convert_ids_to_tokens and convert_tokens_to_string methods of the HuggingFace's tokenizer.

Parameters:

Name Type Description Default
input_ids NDArray[int64]

The input_ids to be decoded

 required
skip_special_tokens bool

Whether to skip the special tokens or not

 required

Returns:

Type Description
List[str]

The decoded texts
Source code in pytorch_widedeep/preprocessing/hf_preprocessor.py 
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
def decode(
    self, input_ids: npt.NDArray[np.int64], skip_special_tokens: bool
) -> List[str]:
    """
    Decodes a list of input_ids. The method is a wrapper around the
    `convert_ids_to_tokens` and `convert_tokens_to_string` methods of the
    HuggingFace's tokenizer.

    Parameters
    ----------
    input_ids: npt.NDArray[np.int64]
        The input_ids to be decoded
    skip_special_tokens: bool
        Whether to skip the special tokens or not

    Returns
    -------
    List[str]
        The decoded texts
    """
    texts = [
        self.tokenizer.convert_tokens_to_string(
            self.tokenizer.convert_ids_to_tokens(input_ids[i], skip_special_tokens)
        )
        for i in range(input_ids.shape[0])
    ]
    return texts

fit 

fit(df)

This method is included for consistency with the rest of the library in general and with the BasePreprocessor in particular. HuggingFace's tokenizers and models are already trained. Therefore, the 'fit' method here does nothing other than checking that the 'text_col' parameter is not None.

Parameters:

Name Type Description Default
df DataFrame

The dataframe containing the text data in the column specified by the 'text_col' parameter

 required
Source code in pytorch_widedeep/preprocessing/hf_preprocessor.py 
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
def fit(self, df: pd.DataFrame) -> "HFPreprocessor":
    """
    This method is included for consistency with the rest of the library
    in general and with the `BasePreprocessor` in particular. HuggingFace's
    tokenizers and models are already trained. Therefore, the 'fit' method
    here does nothing other than checking that the 'text_col' parameter is
    not `None`.

    Parameters
    ----------
    df: pd.DataFrame
        The dataframe containing the text data in the column specified by
        the 'text_col' parameter
    """
    if self.text_col is None:
        raise ValueError(
            "'text_col' is None. Please specify the column name containing the text data"
            " if you want to use the 'fit' method"
        )
    return self

transform 

transform(df)

Encodes the text data in the input dataframe. This method simply calls the encode method under the hood. Similar to the fit method, this method is included for consistency with the rest of the library in general and with the BasePreprocessor in particular.

Parameters:

Name Type Description Default
df DataFrame

The dataframe containing the text data in the column specified by the 'text_col' parameter

 required

Returns:

Type Description
array

The encoded texts
Source code in pytorch_widedeep/preprocessing/hf_preprocessor.py 
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
def transform(self, df: pd.DataFrame) -> npt.NDArray[np.int64]:
    """
    Encodes the text data in the input dataframe. This method simply
    calls the `encode` method under the hood. Similar to the `fit` method,
    this method is included for consistency with the rest of the library
    in general and with the `BasePreprocessor` in particular.

    Parameters
    ----------
    df: pd.DataFrame
        The dataframe containing the text data in the column specified by
        the 'text_col' parameter

    Returns
    -------
    np.array
        The encoded texts
    """
    if self.text_col is None:
        raise ValueError(
            "'text_col' is None. Please specify the column name containing the text data"
            " if you want to use the 'fit' method"
        )

    texts = self._read_texts(df, self.root_dir)

    return self.encode(texts)

transform_sample 

transform_sample(text)

Encodes a single text sample.

Parameters:

Name Type Description Default
text str

The text sample to be encoded

 required

Returns:

Type Description
array

The encoded text
Source code in pytorch_widedeep/preprocessing/hf_preprocessor.py 
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
def transform_sample(self, text: str) -> npt.NDArray[np.int64]:
    """
    Encodes a single text sample.

    Parameters
    ----------
    text: str
        The text sample to be encoded

    Returns
    -------
    np.array
        The encoded text
    """

    if not self.is_fitted:
        raise ValueError(
            "The `encode` (or `fit`) method must be called before calling `transform_sample`"
        )
    return self.encode([text])[0]

fit_transform 

fit_transform(df)

Encodes the text data in the input dataframe.

Parameters:

Name Type Description Default
df DataFrame

The dataframe containing the text data in the column specified by the 'text_col' parameter

 required

Returns:

Type Description
array

The encoded texts
Source code in pytorch_widedeep/preprocessing/hf_preprocessor.py 
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
def fit_transform(self, df: pd.DataFrame) -> npt.NDArray[np.int64]:
    """
    Encodes the text data in the input dataframe.

    Parameters
    ----------
    df: pd.DataFrame
        The dataframe containing the text data in the column specified by
        the 'text_col' parameter

    Returns
    -------
    np.array
        The encoded texts
    """
    return self.fit(df).transform(df)

inverse_transform 

inverse_transform(input_ids, skip_special_tokens)

Decodes a list of input_ids. The method simply calls the decode method under the hood.

Parameters:

Name Type Description Default
input_ids NDArray[int64]

The input_ids to be decoded

 required
skip_special_tokens bool

Whether to skip the special tokens or not

 required

Returns:

Type Description
List[str]

The decoded texts
Source code in pytorch_widedeep/preprocessing/hf_preprocessor.py 
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
def inverse_transform(
    self, input_ids: npt.NDArray[np.int64], skip_special_tokens: bool
) -> List[str]:
    """
    Decodes a list of input_ids. The method simply calls the `decode` method
    under the hood.

    Parameters
    ----------
    input_ids: npt.NDArray[np.int64]
        The input_ids to be decoded
    skip_special_tokens: bool
        Whether to skip the special tokens or not

    Returns
    -------
    List[str]
        The decoded texts
    """
    return self.decode(input_ids, skip_special_tokens)

ImagePreprocessor

Bases: BasePreprocessor

Preprocessor to prepare the deepimage input dataset.

The Preprocessing consists simply on resizing according to their aspect ratio

Parameters:

Name Type Description Default
img_col str

name of the column with the images filenames

 required
img_path str

path to the dicrectory where the images are stored

 required
width int

width of the resulting processed image.

 224
height int

width of the resulting processed image.

 224
verbose int

Enable verbose output.

 1

Attributes:

Name Type Description
aap AspectAwarePreprocessor

an instance of pytorch_widedeep.utils.image_utils.AspectAwarePreprocessor
spp SimplePreprocessor

an instance of pytorch_widedeep.utils.image_utils.SimplePreprocessor
normalise_metrics Dict

Dict containing the normalisation metrics of the image dataset, i.e. mean and std for the R, G and B channels

Examples:

>>> import pandas as pd
>>>
>>> from pytorch_widedeep.preprocessing import ImagePreprocessor
>>>
>>> path_to_image1 = 'tests/test_data_utils/images/galaxy1.png'
>>> path_to_image2 = 'tests/test_data_utils/images/galaxy2.png'
>>>
>>> df_train = pd.DataFrame({'images_column': [path_to_image1]})
>>> df_test = pd.DataFrame({'images_column': [path_to_image2]})
>>> img_preprocessor = ImagePreprocessor(img_col='images_column', img_path='.', verbose=0)
>>> resized_images = img_preprocessor.fit_transform(df_train)
>>> new_resized_images = img_preprocessor.transform(df_train)

ℹ️ NOTE: Normalising metrics will only be computed when the fit_transform method is run. Running transform only will not change the computed metrics and running fit only simply instantiates the resizing functions.

Source code in pytorch_widedeep/preprocessing/image_preprocessor.py 
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
class ImagePreprocessor(BasePreprocessor):
    r"""Preprocessor to prepare the ``deepimage`` input dataset.

    The Preprocessing consists simply on resizing according to their
    aspect ratio

    Parameters
    ----------
    img_col: str
        name of the column with the images filenames
    img_path: str
        path to the dicrectory where the images are stored
    width: int, default=224
        width of the resulting processed image.
    height: int, default=224
        width of the resulting processed image.
    verbose: int, default 1
        Enable verbose output.

    Attributes
    ----------
    aap: AspectAwarePreprocessor
        an instance of `pytorch_widedeep.utils.image_utils.AspectAwarePreprocessor`
    spp: SimplePreprocessor
        an instance of `pytorch_widedeep.utils.image_utils.SimplePreprocessor`
    normalise_metrics: Dict
        Dict containing the normalisation metrics of the image dataset, i.e.
        mean and std for the R, G and B channels

    Examples
    --------
    >>> import pandas as pd
    >>>
    >>> from pytorch_widedeep.preprocessing import ImagePreprocessor
    >>>
    >>> path_to_image1 = 'tests/test_data_utils/images/galaxy1.png'
    >>> path_to_image2 = 'tests/test_data_utils/images/galaxy2.png'
    >>>
    >>> df_train = pd.DataFrame({'images_column': [path_to_image1]})
    >>> df_test = pd.DataFrame({'images_column': [path_to_image2]})
    >>> img_preprocessor = ImagePreprocessor(img_col='images_column', img_path='.', verbose=0)
    >>> resized_images = img_preprocessor.fit_transform(df_train)
    >>> new_resized_images = img_preprocessor.transform(df_train)

    :information_source: **NOTE**:
    Normalising metrics will only be computed when the ``fit_transform``
    method is run. Running ``transform`` only will not change the computed
    metrics and running ``fit`` only simply instantiates the resizing
    functions.
    """

    def __init__(
        self,
        img_col: str,
        img_path: str,
        width: int = 224,
        height: int = 224,
        verbose: int = 1,
    ):
        super(ImagePreprocessor, self).__init__()

        self.img_col = img_col
        self.img_path = img_path
        self.width = width
        self.height = height
        self.verbose = verbose

        self.aap = AspectAwarePreprocessor(self.width, self.height)
        self.spp = SimplePreprocessor(self.width, self.height)

        self.compute_normalising_computed = False

    def fit(self, df: pd.DataFrame) -> BasePreprocessor:
        # simply include this method to comply with the BasePreprocessor and
        # for consistency with the rest of the preprocessors
        return self

    def transform_sample(self, img: np.ndarray) -> np.ndarray:
        aspect_r = img.shape[0] / img.shape[1]
        if aspect_r != 1.0:
            resized_img = self.aap.preprocess(img)
        else:
            resized_img = self.spp.preprocess(img)
        return resized_img

    def transform(self, df: pd.DataFrame) -> np.ndarray:
        """Resizes the images to the input height and width.


        Parameters
        ----------
        df: pd.DataFrame
            Input pandas dataframe with the `img_col`

        Returns
        -------
        np.ndarray
            Resized images to the input height and width
        """
        image_list = df[self.img_col].tolist()
        if self.verbose:
            print("Reading Images from {}".format(self.img_path))
        imgs = [cv2.imread("/".join([self.img_path, img])) for img in image_list]

        # finding images with different height and width
        aspect = [(im.shape[0], im.shape[1]) for im in imgs]
        aspect_r = [a[0] / a[1] for a in aspect]
        diff_idx = [i for i, r in enumerate(aspect_r) if r != 1.0]

        if self.verbose:
            print("Resizing")
        resized_imgs = []
        for i, img in tqdm(enumerate(imgs), total=len(imgs), disable=self.verbose != 1):
            if i in diff_idx:
                resized_imgs.append(self.aap.preprocess(img))
            else:
                # if aspect ratio is 1:1, no need for AspectAwarePreprocessor
                resized_imgs.append(self.spp.preprocess(img))

        if not self.compute_normalising_computed:
            if self.verbose:
                print("Computing normalisation metrics")
            # mean and std deviation will only be computed when the fit method
            # is called
            mean_R, mean_G, mean_B = [], [], []
            std_R, std_G, std_B = [], [], []
            for rsz_img in resized_imgs:
                (mean_b, mean_g, mean_r), (std_b, std_g, std_r) = cv2.meanStdDev(
                    rsz_img
                )
                mean_R.append(mean_r)
                mean_G.append(mean_g)
                mean_B.append(mean_b)
                std_R.append(std_r)
                std_G.append(std_g)
                std_B.append(std_b)
            self.normalise_metrics = dict(
                mean={
                    "R": np.mean(mean_R) / 255.0,
                    "G": np.mean(mean_G) / 255.0,
                    "B": np.mean(mean_B) / 255.0,
                },
                std={
                    "R": np.mean(std_R) / 255.0,
                    "G": np.mean(std_G) / 255.0,
                    "B": np.mean(std_B) / 255.0,
                },
            )
            self.compute_normalising_computed = True
        return np.asarray(resized_imgs)

    def fit_transform(self, df: pd.DataFrame) -> np.ndarray:
        """Combines `fit` and `transform`

        Parameters
        ----------
        df: pd.DataFrame
            Input pandas dataframe

        Returns
        -------
        np.ndarray
            Resized images to the input height and width
        """
        return self.fit(df).transform(df)

    def inverse_transform(self, transformed_image):
        raise NotImplementedError(
            "'inverse_transform' method is not implemented for 'ImagePreprocessor'"
        )

    def __repr__(self) -> str:
        list_of_params: List[str] = []
        list_of_params.append("img_col={img_col}")
        list_of_params.append("img_path={img_path}")
        list_of_params.append("width={width}")
        list_of_params.append("height={height}")
        list_of_params.append("verbose={verbose}")
        all_params = ", ".join(list_of_params)
        return f"ImagePreprocessor({all_params.format(**self.__dict__)})"

transform 

transform(df)

Resizes the images to the input height and width.

Parameters:

Name Type Description Default
df DataFrame

Input pandas dataframe with the img_col

 required

Returns:

Type Description
ndarray

Resized images to the input height and width
Source code in pytorch_widedeep/preprocessing/image_preprocessor.py 
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
def transform(self, df: pd.DataFrame) -> np.ndarray:
    """Resizes the images to the input height and width.


    Parameters
    ----------
    df: pd.DataFrame
        Input pandas dataframe with the `img_col`

    Returns
    -------
    np.ndarray
        Resized images to the input height and width
    """
    image_list = df[self.img_col].tolist()
    if self.verbose:
        print("Reading Images from {}".format(self.img_path))
    imgs = [cv2.imread("/".join([self.img_path, img])) for img in image_list]

    # finding images with different height and width
    aspect = [(im.shape[0], im.shape[1]) for im in imgs]
    aspect_r = [a[0] / a[1] for a in aspect]
    diff_idx = [i for i, r in enumerate(aspect_r) if r != 1.0]

    if self.verbose:
        print("Resizing")
    resized_imgs = []
    for i, img in tqdm(enumerate(imgs), total=len(imgs), disable=self.verbose != 1):
        if i in diff_idx:
            resized_imgs.append(self.aap.preprocess(img))
        else:
            # if aspect ratio is 1:1, no need for AspectAwarePreprocessor
            resized_imgs.append(self.spp.preprocess(img))

    if not self.compute_normalising_computed:
        if self.verbose:
            print("Computing normalisation metrics")
        # mean and std deviation will only be computed when the fit method
        # is called
        mean_R, mean_G, mean_B = [], [], []
        std_R, std_G, std_B = [], [], []
        for rsz_img in resized_imgs:
            (mean_b, mean_g, mean_r), (std_b, std_g, std_r) = cv2.meanStdDev(
                rsz_img
            )
            mean_R.append(mean_r)
            mean_G.append(mean_g)
            mean_B.append(mean_b)
            std_R.append(std_r)
            std_G.append(std_g)
            std_B.append(std_b)
        self.normalise_metrics = dict(
            mean={
                "R": np.mean(mean_R) / 255.0,
                "G": np.mean(mean_G) / 255.0,
                "B": np.mean(mean_B) / 255.0,
            },
            std={
                "R": np.mean(std_R) / 255.0,
                "G": np.mean(std_G) / 255.0,
                "B": np.mean(std_B) / 255.0,
            },
        )
        self.compute_normalising_computed = True
    return np.asarray(resized_imgs)

fit_transform 

fit_transform(df)

Combines fit and transform

Parameters:

Name Type Description Default
df DataFrame

Input pandas dataframe

 required

Returns:

Type Description
ndarray

Resized images to the input height and width
Source code in pytorch_widedeep/preprocessing/image_preprocessor.py 
166
167
168
169
170
171
172
173
174
175
176
177
178
179
def fit_transform(self, df: pd.DataFrame) -> np.ndarray:
    """Combines `fit` and `transform`

    Parameters
    ----------
    df: pd.DataFrame
        Input pandas dataframe

    Returns
    -------
    np.ndarray
        Resized images to the input height and width
    """
    return self.fit(df).transform(df)

DINPreprocessor

Bases: BasePreprocessor

Preprocessor for Deep Interest Network (DIN) models.

This preprocessor handles the preparation of data for DIN models, including sequence building, label encoding, and handling of various types of input columns (categorical, continuous, and sequential).

Parameters:

Name Type Description Default
user_id_col str

Name of the column containing user IDs.

 required
item_embed_col Union[str, Tuple[str, int]]

Name of the column containing item IDs to be embedded, or a tuple of (column_name, embedding_dim). If embedding_dim is not provided, it will be automatically calculated using a rule of thumb specified by embedding_rule. Aliased as item_id_col.

 required
target_col str

Name of the column containing the target variable.

 required
max_seq_length int

Maximum length of sequences to be created.

 required
action_col Optional[str]

Name of the column containing user actions (if applicable), for example a rating, or purchased/not-purchased, etc). This 'action_col' can also be the same as the 'target_col'.

 None
other_seq_embed_cols Optional[List[str] | List[Tuple[str, int]]]

List of other columns to be treated as sequences. Each element in the list can be a string with the name of the column or a tuple with the name of the column and the embedding dimension.

 None
cat_embed_cols Optional[Union[List[str], List[Tuple[str, int]]]]

This and the following parameters are identical to those used in the TabPreprocessor.
List containing the name of the categorical columns that will be represented by embeddings (e.g. ['education', 'relationship', ...]) or a Tuple with the name and the embedding dimension (e.g.: [ ('education',32), ('relationship',16), ...])

 None
continuous_cols Optional[List[str]]

List with the name of the continuous cols

 None
quantization_setup Optional[Union[int, Dict[str, Union[int, List[float]]]]]

Continuous columns can be turned into categorical via pd.cut. If quantization_setup is an int, all continuous columns will be quantized using this value as the number of bins. Alternatively, a dictionary where the keys are the column names to quantize and the values are the either integers indicating the number of bins or a list of scalars indicating the bin edges can also be used.

 None
cols_to_scale Optional[Union[List[str], str]]

List with the names of the columns that will be standarised via sklearn's StandardScaler. It can also be the string 'all' in which case all the continuous cols will be scaled.

 None
auto_embed_dim bool

Boolean indicating whether the embedding dimensions will be automatically defined via rule of thumb. See embedding_rule below.

 True
embedding_rule Literal['google', 'fastai_old', 'fastai_new']

If auto_embed_dim=True, this is the choice of embedding rule of thumb. Choices are:

  • fastai_new: \(min(600, round(1.6 imes n_{cat}^{0.56}))\)

  • fastai_old: \(min(50, (n_{cat}//{2})+1)\)

  • google: \(min(600, round(n_{cat}^{0.24}))\)

 'fastai_new'
default_embed_dim int

Dimension for the embeddings if the embedding dimension is not provided in the cat_embed_cols parameter and auto_embed_dim is set to False.

 16
verbose int

Verbosity level.

 1
scale bool

ℹ️ note: this arg will be removed in upcoming releases. Please use cols_to_scale instead.
Bool indicating whether or not to scale/standarise continuous cols. It is important to emphasize that all the DL models for tabular data in the library also include the possibility of normalising the input continuous features via a BatchNorm or a LayerNorm.
Param alias: scale_cont_cols.

 False
already_standard Optional[List[str]]

ℹ️ note: this arg will be removed in upcoming releases. Please use cols_to_scale instead.
List with the name of the continuous cols that do not need to be scaled/standarised.

 None

Other Parameters:

Name Type Description
**kwargs

pd.cut and StandardScaler related args

Attributes:

Name Type Description
is_fitted bool

Whether the preprocessor has been fitted.
has_standard_tab_data bool

Whether the data includes standard tabular data.
tab_preprocessor TabPreprocessor

Preprocessor for standard tabular data (if applicable).
din_columns_idx Dict[str, int]

Dictionary mapping column names to their indices in the processed data.
item_le LabelEncoder

Label encoder for item IDs.
n_items int

Number of unique items.
user_behaviour_config Tuple[List[str], int, int]

Configuration for user behavior sequences.
action_le LabelEncoder

Label encoder for action column (if applicable).
n_actions int

Number of unique actions (if applicable).
action_seq_config Tuple[List[str], int]

Configuration for action sequences (if applicable).
other_seq_le LabelEncoder

Label encoder for other sequence columns (if applicable).
n_other_seq_cols Dict[str, int]

Number of unique values in each other sequence column.
other_seq_config List[Tuple[List[str], int, int]]

Configuration for other sequence columns.

Examples:

>>> import pandas as pd
>>> from pytorch_widedeep.preprocessing import DINPreprocessor
>>> data = {
...    'user_id': [1, 1, 1, 2, 2, 2, 3, 3, 3],
...    'item_id': [101, 102, 103, 101, 103, 104, 102, 103, 104],
...    'timestamp': [1, 2, 3, 1, 2, 3, 1, 2, 3],
...    'category': ['A', 'B', 'A', 'B', 'A', 'C', 'B', 'A', 'C'],
...    'price': [10.5, 15.0, 12.0, 10.5, 12.0, 20.0, 15.0, 12.0, 20.0],
...    'rating': [0, 1, 0, 1, 0, 1, 0, 1, 0]
... }
>>> df = pd.DataFrame(data)
>>> din_preprocessor = DINPreprocessor(
...     user_id_col='user_id',
...     item_embed_col='item_id',
...     target_col='rating',
...     max_seq_length=2,
...     action_col='rating',
...     other_seq_embed_cols=['category'],
...     cat_embed_cols=['user_id'],
...     continuous_cols=['price'],
...     cols_to_scale=['price']
... )
>>> X, y = din_preprocessor.fit_transform(df)
Source code in pytorch_widedeep/preprocessing/din_preprocessor.py 
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
class DINPreprocessor(BasePreprocessor):
    """
    Preprocessor for Deep Interest Network (DIN) models.

    This preprocessor handles the preparation of data for DIN models,
    including sequence building, label encoding, and handling of various
    types of input columns (categorical, continuous, and sequential).

    Parameters
    ----------
    user_id_col : str
        Name of the column containing user IDs.
    item_embed_col : Union[str, Tuple[str, int]]
        Name of the column containing item IDs to be embedded, or a tuple of
        (column_name, embedding_dim). If embedding_dim is not provided, it
        will be automatically calculated using a rule of thumb specified by
        `embedding_rule`. Aliased as `item_id_col`.
    target_col : str
        Name of the column containing the target variable.
    max_seq_length : int
        Maximum length of sequences to be created.
    action_col : Optional[str], default=None
        Name of the column containing user actions (if applicable), for
        example a rating, or purchased/not-purchased, etc). This 'action_col'
        can also be the same as the 'target_col'.
    other_seq_embed_cols : Optional[List[str] | List[Tuple[str, int]]], default=None
        List of other columns to be treated as sequences. Each element in the
        list can be a string with the name of the column or a tuple with the
        name of the column and the embedding dimension.

    cat_embed_cols : Optional[Union[List[str], List[Tuple[str, int]]]], default=None
        This and the following parameters are identical to those used in the
        TabPreprocessor.<br>
        List containing the name of the categorical columns that will be
        represented by embeddings (e.g. _['education', 'relationship', ...]_) or
        a Tuple with the name and the embedding dimension (e.g.: _[
        ('education',32), ('relationship',16), ...]_)
    continuous_cols: List, default = None
        List with the name of the continuous cols
    quantization_setup: int or Dict, default = None
        Continuous columns can be turned into categorical via `pd.cut`. If
        `quantization_setup` is an `int`, all continuous columns will be
        quantized using this value as the number of bins. Alternatively, a
        dictionary where the keys are the column names to quantize and the
        values are the either integers indicating the number of bins or a
        list of scalars indicating the bin edges can also be used.
    cols_to_scale: List or str, default = None,
        List with the names of the columns that will be standarised via
        sklearn's `StandardScaler`. It can also be the string `'all'` in
        which case all the continuous cols will be scaled.
    auto_embed_dim: bool, default = True
        Boolean indicating whether the embedding dimensions will be
        automatically defined via rule of thumb. See `embedding_rule`
        below.
    embedding_rule: str, default = 'fastai_new'
        If `auto_embed_dim=True`, this is the choice of embedding rule of
        thumb. Choices are:

        - _fastai_new_: $min(600, round(1.6 \times n_{cat}^{0.56}))$

        - _fastai_old_: $min(50, (n_{cat}//{2})+1)$

        - _google_: $min(600, round(n_{cat}^{0.24}))$
    default_embed_dim: int, default=16
        Dimension for the embeddings if the embedding dimension is not
        provided in the `cat_embed_cols` parameter and `auto_embed_dim` is
        set to `False`.
    verbose : int, default=1
        Verbosity level.
    scale: bool, default = False
        :information_source: **note**: this arg will be removed in upcoming
         releases. Please use `cols_to_scale` instead. <br/> Bool indicating
         whether or not to scale/standarise continuous cols. It is important
         to emphasize that all the DL models for tabular data in the library
         also include the possibility of normalising the input continuous
         features via a `BatchNorm` or a `LayerNorm`. <br/> Param alias:
         `scale_cont_cols`.
    already_standard: List, default = None
        :information_source: **note**: this arg will be removed in upcoming
         releases. Please use `cols_to_scale` instead. <br/> List with the
         name of the continuous cols that do not need to be
         scaled/standarised.

    Other Parameters
    ----------------
    **kwargs: dict
        `pd.cut` and `StandardScaler` related args

    Attributes
    ----------
    is_fitted : bool
        Whether the preprocessor has been fitted.
    has_standard_tab_data : bool
        Whether the data includes standard tabular data.
    tab_preprocessor : TabPreprocessor
        Preprocessor for standard tabular data (if applicable).
    din_columns_idx : Dict[str, int]
        Dictionary mapping column names to their indices in the processed data.
    item_le : LabelEncoder
        Label encoder for item IDs.
    n_items : int
        Number of unique items.
    user_behaviour_config : Tuple[List[str], int, int]
        Configuration for user behavior sequences.
    action_le : LabelEncoder
        Label encoder for action column (if applicable).
    n_actions : int
        Number of unique actions (if applicable).
    action_seq_config : Tuple[List[str], int]
        Configuration for action sequences (if applicable).
    other_seq_le : LabelEncoder
        Label encoder for other sequence columns (if applicable).
    n_other_seq_cols : Dict[str, int]
        Number of unique values in each other sequence column.
    other_seq_config : List[Tuple[List[str], int, int]]
        Configuration for other sequence columns.

    Examples
    --------
    >>> import pandas as pd
    >>> from pytorch_widedeep.preprocessing import DINPreprocessor
    >>> data = {
    ...    'user_id': [1, 1, 1, 2, 2, 2, 3, 3, 3],
    ...    'item_id': [101, 102, 103, 101, 103, 104, 102, 103, 104],
    ...    'timestamp': [1, 2, 3, 1, 2, 3, 1, 2, 3],
    ...    'category': ['A', 'B', 'A', 'B', 'A', 'C', 'B', 'A', 'C'],
    ...    'price': [10.5, 15.0, 12.0, 10.5, 12.0, 20.0, 15.0, 12.0, 20.0],
    ...    'rating': [0, 1, 0, 1, 0, 1, 0, 1, 0]
    ... }
    >>> df = pd.DataFrame(data)
    >>> din_preprocessor = DINPreprocessor(
    ...     user_id_col='user_id',
    ...     item_embed_col='item_id',
    ...     target_col='rating',
    ...     max_seq_length=2,
    ...     action_col='rating',
    ...     other_seq_embed_cols=['category'],
    ...     cat_embed_cols=['user_id'],
    ...     continuous_cols=['price'],
    ...     cols_to_scale=['price']
    ... )
    >>> X, y = din_preprocessor.fit_transform(df)
    """

    @alias("item_embed_col", ["item_id_col"])
    def __init__(
        self,
        *,
        user_id_col: str,
        item_embed_col: Union[str, Tuple[str, int]],
        target_col: str,
        max_seq_length: int,
        action_col: Optional[str] = None,
        other_seq_embed_cols: Optional[Union[List[str], List[Tuple[str, int]]]] = None,
        cat_embed_cols: Optional[Union[List[str], List[Tuple[str, int]]]] = None,
        continuous_cols: Optional[List[str]] = None,
        quantization_setup: Optional[
            Union[int, Dict[str, Union[int, List[float]]]]
        ] = None,
        cols_to_scale: Optional[Union[List[str], str]] = None,
        auto_embed_dim: bool = True,
        embedding_rule: Literal["google", "fastai_old", "fastai_new"] = "fastai_new",
        default_embed_dim: int = 16,
        verbose: int = 1,
        scale: bool = False,
        already_standard: Optional[List[str]] = None,
        **kwargs,
    ):
        self.user_id_col = user_id_col
        self.item_embed_col = item_embed_col
        self.max_seq_length = max_seq_length
        self.target_col = target_col if target_col is not None else "target"
        self.action_col = action_col
        self.other_seq_embed_cols = other_seq_embed_cols
        self.cat_embed_cols = cat_embed_cols
        self.continuous_cols = continuous_cols
        self.quantization_setup = quantization_setup
        self.cols_to_scale = cols_to_scale
        self.auto_embed_dim = auto_embed_dim
        self.embedding_rule = embedding_rule
        self.default_embed_dim = default_embed_dim
        self.verbose = verbose
        self.scale = scale
        self.already_standard = already_standard
        self.kwargs = kwargs

        self.has_standard_tab_data = bool(self.cat_embed_cols or self.continuous_cols)
        if self.has_standard_tab_data:
            self.tab_preprocessor = TabPreprocessor(
                cat_embed_cols=self.cat_embed_cols,
                continuous_cols=self.continuous_cols,
                quantization_setup=self.quantization_setup,
                cols_to_scale=self.cols_to_scale,
                auto_embed_dim=self.auto_embed_dim,
                embedding_rule=self.embedding_rule,
                default_embed_dim=self.default_embed_dim,
                verbose=self.verbose,
                scale=self.scale,
                already_standard=self.already_standard,
                **self.kwargs,
            )

        self.is_fitted = False

    def fit(self, df: pd.DataFrame) -> "DINPreprocessor":
        if self.has_standard_tab_data:
            self.tab_preprocessor.fit(df)
            self.din_columns_idx = {
                col: i for i, col in enumerate(self.tab_preprocessor.column_idx.keys())
            }
        else:
            self.din_columns_idx = {}

        self._fit_label_encoders(df)
        self.is_fitted = True
        return self

    def transform(self, df: pd.DataFrame) -> Tuple[np.ndarray, np.ndarray]:
        _df = self._pre_transform(df)
        df_w_sequences = self._build_sequences(_df)
        X_all = self._concatenate_features(df_w_sequences)
        return X_all, np.array(df_w_sequences[self.target_col].tolist())

    def fit_transform(self, df: pd.DataFrame) -> Tuple[np.ndarray, np.ndarray]:
        self.fit(df)
        return self.transform(df)

    def inverse_transform(self, X: np.ndarray) -> pd.DataFrame:
        raise NotImplementedError(
            "inverse_transform is not implemented for this preprocessor"
        )

    def _fit_label_encoders(self, df: pd.DataFrame) -> None:
        self.item_le = LabelEncoder(columns_to_encode=[self._get_item_col()])
        self.item_le.fit(df)
        self.n_items = max(self.item_le.encoding_dict[self._get_item_col()].values())
        user_behaviour_embed_size = self._get_embed_size(
            self.item_embed_col, self.n_items
        )
        self.user_behaviour_config = (
            [f"item_{i+1}" for i in range(self.max_seq_length)],
            self.n_items,
            user_behaviour_embed_size,
        )

        self._update_din_columns_idx("item", self.max_seq_length)
        self.din_columns_idx["target_item"] = len(self.din_columns_idx)

        if self.action_col:
            self._fit_action_label_encoder(df)
        if self.other_seq_embed_cols:
            self._fit_other_seq_label_encoders(df)

    def _fit_action_label_encoder(self, df: pd.DataFrame) -> None:
        self.action_le = LabelEncoder(columns_to_encode=[self.action_col])
        self.action_le.fit(df)
        self.n_actions = len(self.action_le.encoding_dict[self.action_col])
        self.action_seq_config = (
            [f"action_{i+1}" for i in range(self.max_seq_length)],
            self.n_actions,
        )
        self._update_din_columns_idx("action", self.max_seq_length)

    def _other_seq_cols_float_warning(self, df: pd.DataFrame) -> None:
        other_seq_cols = [
            col[0] if isinstance(col, tuple) else col
            for col in self.other_seq_embed_cols
        ]
        for col in other_seq_cols:
            if df[col].dtype == float:
                warnings.warn(
                    f"{col} is a float column. It will be converted to integers. "
                    "If this is not what you want, please convert it beforehand.",
                    UserWarning,
                )

    def _convert_other_seq_cols_to_int(self, df: pd.DataFrame) -> None:
        other_seq_cols = [
            col[0] if isinstance(col, tuple) else col
            for col in self.other_seq_embed_cols
        ]
        for col in other_seq_cols:
            if df[col].dtype == float:
                df[col] = df[col].astype(int)

    def _fit_other_seq_label_encoders(self, df: pd.DataFrame) -> None:
        self._other_seq_cols_float_warning(df)
        self._convert_other_seq_cols_to_int(df)
        self.other_seq_le = LabelEncoder(
            columns_to_encode=[
                col[0] if isinstance(col, tuple) else col
                for col in self.other_seq_embed_cols
            ]
        )
        self.other_seq_le.fit(df)
        other_seq_cols = [
            col[0] if isinstance(col, tuple) else col
            for col in self.other_seq_embed_cols
        ]
        self.n_other_seq_cols = {
            col: max(self.other_seq_le.encoding_dict[col].values())
            for col in other_seq_cols
        }
        other_seq_embed_sizes = {
            col: self._get_embed_size(col, self.n_other_seq_cols[col])
            for col in other_seq_cols
        }
        self.other_seq_config = [
            (
                self._get_seq_col_names(col),
                self.n_other_seq_cols[col],
                other_seq_embed_sizes[col],
            )
            for col in other_seq_cols
        ]
        self._update_din_columns_idx_for_other_seq(other_seq_cols)

    def _get_item_col(self) -> str:
        return (
            self.item_embed_col[0]
            if isinstance(self.item_embed_col, tuple)
            else self.item_embed_col
        )

    def _get_embed_size(self, col: Union[str, Tuple[str, int]], n_unique: int) -> int:
        return col[1] if isinstance(col, tuple) else embed_sz_rule(n_unique)

    def _get_seq_col_names(self, col: str) -> List[str]:
        return [f"{col}_{i+1}" for i in range(self.max_seq_length)]

    def _update_din_columns_idx(self, prefix: str, length: int) -> None:
        current_len = len(self.din_columns_idx)
        self.din_columns_idx.update(
            {f"{prefix}_{i+1}": i + current_len for i in range(length)}
        )

    def _update_din_columns_idx_for_other_seq(self, other_seq_cols: List[str]) -> None:
        current_len = len(self.din_columns_idx)
        for col in other_seq_cols:
            self.din_columns_idx.update(
                {f"{col}_{i+1}": i + current_len for i in range(self.max_seq_length)}
            )
            self.din_columns_idx[f"target_{col}"] = len(self.din_columns_idx)
            current_len = len(self.din_columns_idx)

    def _pre_transform(self, df: pd.DataFrame) -> pd.DataFrame:
        check_is_fitted(self, attributes=["din_columns_idx"])
        df = self.item_le.transform(df)
        if self.action_col:
            df = self.action_le.transform(df)
        if self.other_seq_embed_cols:
            df = self.other_seq_le.transform(df)
        return df

    def _build_sequences(self, df: pd.DataFrame) -> pd.DataFrame:
        # df.columns.tolist() -> to avoid annoying pandas warning
        return (
            df.groupby(self.user_id_col)[df.columns.tolist()]  # type: ignore[index]
            .apply(self._group_sequences)
            .reset_index(drop=True)
        )

    def _group_sequences(self, group: pd.DataFrame) -> pd.DataFrame:
        item_col = self._get_item_col()
        items = group[item_col].tolist()
        targets = group[self.target_col].tolist()
        drop_cols = [item_col, self.target_col]

        # sequences cannot be built with user, item pairs with only one
        # interaction
        if len(items) <= 1:
            return pd.DataFrame()

        sequences = [
            self._create_sequence(group, items, targets, i, drop_cols)
            for i in range(max(1, len(items) - self.max_seq_length))
        ]

        seq_df = pd.DataFrame(sequences)
        non_seq_cols = group.drop_duplicates([self.user_id_col]).drop(drop_cols, axis=1)
        return pd.merge(seq_df, non_seq_cols, on=self.user_id_col)

    def _create_sequence(
        self,
        group: pd.DataFrame,
        items: List[int],
        targets: List[int],
        i: int,
        drop_cols: List[str],
    ) -> Dict[str, Union[str, List[int]]]:
        end_idx = min(i + self.max_seq_length, len(items) - 1)
        item_sequences = items[i:end_idx]
        target_item = items[end_idx]
        target = targets[end_idx]

        sequence = {
            self.user_id_col: group.name,
            "items_sequence": item_sequences,
            "target_item": target_item,
        }

        if self.action_col:
            sequence.update(
                self._create_action_sequence(group, targets, i, target, drop_cols)
            )
        else:
            sequence[self.target_col] = target
            drop_cols.append(self.target_col)

        if self.other_seq_embed_cols:
            sequence.update(self._create_other_seq_sequence(group, i, drop_cols))

        return sequence

    def _create_action_sequence(
        self,
        group: pd.DataFrame,
        targets: List[int],
        i: int,
        target: int,
        drop_cols: List[str],
    ) -> Dict[str, Union[int, List[int]]]:
        if self.action_col != self.target_col:
            actions = group[self.action_col].tolist()
            action_sequences = (
                actions[i : i + self.max_seq_length]
                if i + self.max_seq_length < len(actions)
                else actions[i:-1]
            )
            drop_cols.append(self.action_col)
        else:
            target -= 1  # the 'transform' method adds 1 as it saves 0 for padding
            action_sequences = (
                targets[i : i + self.max_seq_length]
                if i + self.max_seq_length < len(targets)
                else targets[i:-1]
            )

        return {self.target_col: target, "actions_sequence": action_sequences}

    def _create_other_seq_sequence(
        self, group: pd.DataFrame, i: int, drop_cols: List[str]
    ) -> Dict[str, Union[int, List[int]]]:
        other_seq_cols = [
            col[0] if isinstance(col, tuple) else col
            for col in self.other_seq_embed_cols
        ]
        drop_cols += other_seq_cols
        other_seqs = {col: group[col].tolist() for col in other_seq_cols}

        other_seqs_sequences: Dict[str, Union[int, List[int]]] = {}
        for col in other_seq_cols:
            other_seqs_sequences[col] = (
                other_seqs[col][i : i + self.max_seq_length]
                if i + self.max_seq_length < len(other_seqs[col])
                else other_seqs[col][i:-1]
            )

        sequence: Dict[str, Union[int, List[int]]] = {}
        for col in other_seq_cols:
            sequence[f"{col}_sequence"] = other_seqs_sequences[col]
            sequence[f"target_{col}"] = (
                other_seqs[col][i + self.max_seq_length]
                if i + self.max_seq_length < len(other_seqs[col])
                else other_seqs[col][-1]
            )

        return sequence

    def _concatenate_features(self, df_w_sequences: pd.DataFrame) -> np.ndarray:
        user_behaviour_seq = df_w_sequences["items_sequence"].tolist()
        X_user_behaviour = np.vstack(
            [
                pad_sequences(seq, self.max_seq_length, pad_idx=0)
                for seq in user_behaviour_seq
            ]
        )
        X_target_item = np.array(df_w_sequences["target_item"].tolist()).reshape(-1, 1)
        X_all = np.concatenate([X_user_behaviour, X_target_item], axis=1)

        if self.has_standard_tab_data:
            X_tab = self.tab_preprocessor.transform(df_w_sequences)
            X_all = np.concatenate([X_tab, X_all], axis=1)

        if self.action_col:
            action_seq = df_w_sequences["actions_sequence"].tolist()
            X_actions = np.vstack(
                [
                    pad_sequences(seq, self.max_seq_length, pad_idx=0)
                    for seq in action_seq
                ]
            )
            X_all = np.concatenate([X_all, X_actions], axis=1)

        if self.other_seq_embed_cols:
            X_all = self._concatenate_other_seq_features(df_w_sequences, X_all)

        assert len(self.din_columns_idx) == X_all.shape[1], (
            f"Something went wrong. The number of columns in the final array "
            f"({X_all.shape[1]}) is different from the number of columns in "
            f"self.din_columns_idx ({len(self.din_columns_idx)})"
        )

        return X_all

    def _concatenate_other_seq_features(
        self, df_w_sequences: pd.DataFrame, X_all: np.ndarray
    ) -> np.ndarray:
        other_seq_cols = [
            col[0] if isinstance(col, tuple) else col
            for col in self.other_seq_embed_cols
        ]
        other_seq = {
            col: df_w_sequences[f"{col}_sequence"].tolist() for col in other_seq_cols
        }
        other_seq_target = {
            col: df_w_sequences[f"target_{col}"].tolist() for col in other_seq_cols
        }
        X_other_seq_arrays = []

        for col in other_seq_cols:
            X_other_seq_arrays.append(
                np.vstack(
                    [
                        pad_sequences(s, self.max_seq_length, pad_idx=0)
                        for s in other_seq[col]
                    ]
                )
            )
            X_other_seq_arrays.append(np.array(other_seq_target[col]).reshape(-1, 1))

        return np.concatenate([X_all] + X_other_seq_arrays, axis=1)

Chunked versions

Chunked versions of the preprocessors are also available. These are useful when the data is too big to fit in memory. See also the load_from_folder module in the library and the corresponding section here in the documentation.

Note that there is not a ChunkImagePreprocessor. This is because the processing of the images will occur inside the ImageFromFolder class in the load_from_folder module.

ChunkWidePreprocessor

Bases: WidePreprocessor

Preprocessor to prepare the wide input dataset

This Preprocessor prepares the data for the wide, linear component. This linear model is implemented via an Embedding layer that is connected to the output neuron. ChunkWidePreprocessor numerically encodes all the unique values of all categorical columns wide_cols + crossed_cols. See the Example below.

Parameters:

Name Type Description Default
wide_cols List[str]

List of strings with the name of the columns that will label encoded and passed through the wide component

 required
crossed_cols Optional[List[Tuple[str, str]]]

List of Tuples with the name of the columns that will be 'crossed' and then label encoded. e.g. [('education', 'occupation'), ...]. For binary features, a cross-product transformation is 1 if and only if the constituent features are all 1, and 0 otherwise.

 None

Attributes:

Name Type Description
wide_crossed_cols List

List with the names of all columns that will be label encoded
encoding_dict Dict

Dictionary where the keys are the result of pasting colname + '_' + column value and the values are the corresponding mapped integer.
inverse_encoding_dict Dict

the inverse encoding dictionary
wide_dim int

Dimension of the wide model (i.e. dim of the linear layer)

Examples:

>>> import pandas as pd
>>> from pytorch_widedeep.preprocessing import ChunkWidePreprocessor
>>> chunk = pd.DataFrame({'color': ['r', 'b', 'g'], 'size': ['s', 'n', 'l']})
>>> wide_cols = ['color']
>>> crossed_cols = [('color', 'size')]
>>> chunk_wide_preprocessor = ChunkWidePreprocessor(wide_cols=wide_cols, crossed_cols=crossed_cols,
... n_chunks=1)
>>> X_wide = chunk_wide_preprocessor.fit_transform(chunk)
Source code in pytorch_widedeep/preprocessing/wide_preprocessor.py 
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
class ChunkWidePreprocessor(WidePreprocessor):
    r"""Preprocessor to prepare the wide input dataset

    This Preprocessor prepares the data for the wide, linear component.
    This linear model is implemented via an Embedding layer that is
    connected to the output neuron. `ChunkWidePreprocessor` numerically
    encodes all the unique values of all categorical columns `wide_cols +
    crossed_cols`. See the Example below.

    Parameters
    ----------
    wide_cols: List
        List of strings with the name of the columns that will label
        encoded and passed through the `wide` component
    crossed_cols: List, default = None
        List of Tuples with the name of the columns that will be `'crossed'`
        and then label encoded. e.g. _[('education', 'occupation'), ...]_. For
        binary features, a cross-product transformation is 1 if and only if
        the constituent features are all 1, and 0 otherwise.

    Attributes
    ----------
    wide_crossed_cols: List
        List with the names of all columns that will be label encoded
    encoding_dict: Dict
        Dictionary where the keys are the result of pasting `colname + '_' +
        column value` and the values are the corresponding mapped integer.
    inverse_encoding_dict: Dict
        the inverse encoding dictionary
    wide_dim: int
        Dimension of the wide model (i.e. dim of the linear layer)

    Examples
    --------
    >>> import pandas as pd
    >>> from pytorch_widedeep.preprocessing import ChunkWidePreprocessor
    >>> chunk = pd.DataFrame({'color': ['r', 'b', 'g'], 'size': ['s', 'n', 'l']})
    >>> wide_cols = ['color']
    >>> crossed_cols = [('color', 'size')]
    >>> chunk_wide_preprocessor = ChunkWidePreprocessor(wide_cols=wide_cols, crossed_cols=crossed_cols,
    ... n_chunks=1)
    >>> X_wide = chunk_wide_preprocessor.fit_transform(chunk)
    """

    def __init__(
        self,
        wide_cols: List[str],
        n_chunks: int,
        crossed_cols: Optional[List[Tuple[str, str]]] = None,
    ):
        super(ChunkWidePreprocessor, self).__init__(wide_cols, crossed_cols)

        self.n_chunks = n_chunks

        self.chunk_counter = 0

        self.is_fitted = False

    def partial_fit(self, chunk: pd.DataFrame) -> "ChunkWidePreprocessor":
        r"""Fits the Preprocessor and creates required attributes

        Parameters
        ----------
        chunk: pd.DataFrame
            Input pandas dataframe

        Returns
        -------
        ChunkWidePreprocessor
            `ChunkWidePreprocessor` fitted object
        """
        df_wide = self._prepare_wide(chunk)
        self.wide_crossed_cols = df_wide.columns.tolist()

        if self.chunk_counter == 0:
            self.glob_feature_set = set(
                self._make_global_feature_list(df_wide[self.wide_crossed_cols])
            )
        else:
            self.glob_feature_set.update(
                self._make_global_feature_list(df_wide[self.wide_crossed_cols])
            )

        self.chunk_counter += 1

        if self.chunk_counter == self.n_chunks:
            self.encoding_dict = {v: i + 1 for i, v in enumerate(self.glob_feature_set)}
            self.wide_dim = len(self.encoding_dict)
            self.inverse_encoding_dict = {k: v for v, k in self.encoding_dict.items()}
            self.inverse_encoding_dict[0] = "unseen"

            self.is_fitted = True

        return self

    def fit(self, df: pd.DataFrame) -> "ChunkWidePreprocessor":
        """
        Runs `partial_fit`. This is just to override the fit method in the base
        class. This class is not designed or thought to run fit
        """
        return self.partial_fit(df)

    def __repr__(self) -> str:
        list_of_params: List[str] = ["wide_cols={wide_cols}"]
        list_of_params.append("n_chunks={n_chunks}")
        if self.crossed_cols is not None:
            list_of_params.append("crossed_cols={crossed_cols}")
        all_params = ", ".join(list_of_params)
        return f"WidePreprocessor({all_params.format(**self.__dict__)})"

partial_fit 

partial_fit(chunk)

Fits the Preprocessor and creates required attributes

Parameters:

Name Type Description Default
chunk DataFrame

Input pandas dataframe

 required

Returns:

Type Description
ChunkWidePreprocessor

ChunkWidePreprocessor fitted object
Source code in pytorch_widedeep/preprocessing/wide_preprocessor.py 
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
def partial_fit(self, chunk: pd.DataFrame) -> "ChunkWidePreprocessor":
    r"""Fits the Preprocessor and creates required attributes

    Parameters
    ----------
    chunk: pd.DataFrame
        Input pandas dataframe

    Returns
    -------
    ChunkWidePreprocessor
        `ChunkWidePreprocessor` fitted object
    """
    df_wide = self._prepare_wide(chunk)
    self.wide_crossed_cols = df_wide.columns.tolist()

    if self.chunk_counter == 0:
        self.glob_feature_set = set(
            self._make_global_feature_list(df_wide[self.wide_crossed_cols])
        )
    else:
        self.glob_feature_set.update(
            self._make_global_feature_list(df_wide[self.wide_crossed_cols])
        )

    self.chunk_counter += 1

    if self.chunk_counter == self.n_chunks:
        self.encoding_dict = {v: i + 1 for i, v in enumerate(self.glob_feature_set)}
        self.wide_dim = len(self.encoding_dict)
        self.inverse_encoding_dict = {k: v for v, k in self.encoding_dict.items()}
        self.inverse_encoding_dict[0] = "unseen"

        self.is_fitted = True

    return self

fit 

fit(df)

Runs partial_fit. This is just to override the fit method in the base class. This class is not designed or thought to run fit

Source code in pytorch_widedeep/preprocessing/wide_preprocessor.py 
304
305
306
307
308
309
def fit(self, df: pd.DataFrame) -> "ChunkWidePreprocessor":
    """
    Runs `partial_fit`. This is just to override the fit method in the base
    class. This class is not designed or thought to run fit
    """
    return self.partial_fit(df)

ChunkTabPreprocessor

Bases: TabPreprocessor

Preprocessor to prepare the deeptabular component input dataset

Parameters:

Name Type Description Default
n_chunks int

Number of chunks that the tabular dataset is divided by.

 required
cat_embed_cols Optional[Union[List[str], List[Tuple[str, int]]]]

List containing the name of the categorical columns that will be represented by embeddings (e.g. ['education', 'relationship', ...]) or a Tuple with the name and the embedding dimension (e.g.: [ ('education',32), ('relationship',16), ...]).
Param alias: embed_cols

 None
continuous_cols Optional[List[str]]

List with the name of the continuous cols

 None
cols_and_bins Optional[Dict[str, List[float]]]

Continuous columns can be turned into categorical via pd.cut. 'cols_and_bins' is dictionary where the keys are the column names to quantize and the values are a list of scalars indicating the bin edges.
Param alias: quantization_setup

 None
cols_to_scale Optional[Union[List[str], str]]

List with the names of the columns that will be standarised via sklearn's StandardScaler

 None
default_embed_dim int

Dimension for the embeddings if the embed_dim is not provided in the cat_embed_cols parameter and auto_embed_dim is set to False.

 16
with_attention bool

Boolean indicating whether the preprocessed data will be passed to an attention-based model (more precisely a model where all embeddings must have the same dimensions). If True, the param cat_embed_cols must just be a list containing just the categorical column names: e.g. ['education', 'relationship', ...]. This is because they will all be encoded using embeddings of the same dim, which will be specified later when the model is defined.
Param alias: for_transformer

 False
with_cls_token bool

Boolean indicating if a '[CLS]' token will be added to the dataset when using attention-based models. The final hidden state corresponding to this token is used as the aggregated representation for classification and regression tasks. If not, the categorical (and continuous embeddings if present) will be concatenated before being passed to the final MLP (if present).

 False
shared_embed bool

Boolean indicating if the embeddings will be "shared" when using attention-based models. The idea behind shared_embed is described in the Appendix A in the TabTransformer paper: 'The goal of having column embedding is to enable the model to distinguish the classes in one column from those in the other columns'. In other words, the idea is to let the model learn which column is embedded at the time. See: pytorch_widedeep.models.transformers._layers.SharedEmbeddings.

 False
verbose int
1
scale bool

ℹ️ note: this arg will be removed in upcoming releases. Please use cols_to_scale instead.
Bool indicating whether or not to scale/standarise continuous cols. It is important to emphasize that all the DL models for tabular data in the library also include the possibility of normalising the input continuous features via a BatchNorm or a LayerNorm.
Param alias: scale_cont_cols.

 False
already_standard Optional[List[str]]

ℹ️ note: this arg will be removed in upcoming releases. Please use cols_to_scale instead.
List with the name of the continuous cols that do not need to be scaled/standarised.

 None

Other Parameters:

Name Type Description
**kwargs

pd.cut and StandardScaler related args

Attributes:

Name Type Description
cat_cols List[str]

List containing the names of the categorical columns after processing.
cols_and_bins Optional[Dict[str, Union[int, List[float]]]]

Dictionary containing the quantization setup after processing if quantization is requested. This is derived from the quantization_setup parameter.
quant_args Dict

Dictionary containing arguments passed to pandas.cut() function for quantization.
scale_args Dict

Dictionary containing arguments passed to StandardScaler.
label_encoder LabelEncoder

Instance of LabelEncoder used to encode categorical variables. See pytorch_widedeep.utils.dense_utils.LabelEncoder.
cat_embed_input List

List of tuples with the column name, number of individual values for that column and, if with_attention is set to False, the corresponding embeddings dim, e.g. [('education', 16, 10), ('relationship', 6, 8), ...].
standardize_cols List

List of the columns that will be standardized.
scaler StandardScaler

An instance of sklearn.preprocessing.StandardScaler if standardization is requested.
column_idx Dict

Dictionary where keys are column names and values are column indexes. This is necessary to slice tensors.
quantizer Quantizer

An instance of Quantizer if quantization is requested.
is_fitted bool

Boolean indicating if the preprocessor has been fitted.

Examples:

>>> import pandas as pd
>>> import numpy as np
>>> from pytorch_widedeep.preprocessing import ChunkTabPreprocessor
>>> np.random.seed(42)
>>> chunk_df = pd.DataFrame({'cat_col': np.random.choice(['A', 'B', 'C'], size=8),
... 'cont_col': np.random.uniform(1, 100, size=8)})
>>> cat_embed_cols = [('cat_col',4)]
>>> cont_cols = ['cont_col']
>>> tab_preprocessor = ChunkTabPreprocessor(
... n_chunks=1, cat_embed_cols=cat_embed_cols, continuous_cols=cont_cols
... )
>>> X_tab = tab_preprocessor.fit_transform(chunk_df)
>>> tab_preprocessor.cat_embed_cols
[('cat_col', 4)]
>>> tab_preprocessor.column_idx
{'cat_col': 0, 'cont_col': 1}
Source code in pytorch_widedeep/preprocessing/tab_preprocessor.py 
 714
 715
 716
 717
 718
 719
 720
 721
 722
 723
 724
 725
 726
 727
 728
 729
 730
 731
 732
 733
 734
 735
 736
 737
 738
 739
 740
 741
 742
 743
 744
 745
 746
 747
 748
 749
 750
 751
 752
 753
 754
 755
 756
 757
 758
 759
 760
 761
 762
 763
 764
 765
 766
 767
 768
 769
 770
 771
 772
 773
 774
 775
 776
 777
 778
 779
 780
 781
 782
 783
 784
 785
 786
 787
 788
 789
 790
 791
 792
 793
 794
 795
 796
 797
 798
 799
 800
 801
 802
 803
 804
 805
 806
 807
 808
 809
 810
 811
 812
 813
 814
 815
 816
 817
 818
 819
 820
 821
 822
 823
 824
 825
 826
 827
 828
 829
 830
 831
 832
 833
 834
 835
 836
 837
 838
 839
 840
 841
 842
 843
 844
 845
 846
 847
 848
 849
 850
 851
 852
 853
 854
 855
 856
 857
 858
 859
 860
 861
 862
 863
 864
 865
 866
 867
 868
 869
 870
 871
 872
 873
 874
 875
 876
 877
 878
 879
 880
 881
 882
 883
 884
 885
 886
 887
 888
 889
 890
 891
 892
 893
 894
 895
 896
 897
 898
 899
 900
 901
 902
 903
 904
 905
 906
 907
 908
 909
 910
 911
 912
 913
 914
 915
 916
 917
 918
 919
 920
 921
 922
 923
 924
 925
 926
 927
 928
 929
 930
 931
 932
 933
 934
 935
 936
 937
 938
 939
 940
 941
 942
 943
 944
 945
 946
 947
 948
 949
 950
 951
 952
 953
 954
 955
 956
 957
 958
 959
 960
 961
 962
 963
 964
 965
 966
 967
 968
 969
 970
 971
 972
 973
 974
 975
 976
 977
 978
 979
 980
 981
 982
 983
 984
 985
 986
 987
 988
 989
 990
 991
 992
 993
 994
 995
 996
 997
 998
 999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
class ChunkTabPreprocessor(TabPreprocessor):
    r"""Preprocessor to prepare the `deeptabular` component input dataset

    Parameters
    ----------
    n_chunks: int
        Number of chunks that the tabular dataset is divided by.
    cat_embed_cols: List, default = None
        List containing the name of the categorical columns that will be
        represented by embeddings (e.g. _['education', 'relationship', ...]_) or
        a Tuple with the name and the embedding dimension (e.g.: _[
        ('education',32), ('relationship',16), ...]_). <br/> Param alias:
        `embed_cols`
    continuous_cols: List, default = None
        List with the name of the continuous cols
    cols_and_bins: Dict, default = None
        Continuous columns can be turned into categorical via
        `pd.cut`. 'cols_and_bins' is dictionary where the keys are the column
        names to quantize and the values are a list of scalars indicating the
        bin edges. <br/> Param alias: `quantization_setup`
    cols_to_scale: List, default = None,
        List with the names of the columns that will be standarised via
        sklearn's `StandardScaler`
    default_embed_dim: int, default=16
        Dimension for the embeddings if the embed_dim is not provided in the
        `cat_embed_cols` parameter and `auto_embed_dim` is set to
        `False`.
    with_attention: bool, default = False
        Boolean indicating whether the preprocessed data will be passed to an
        attention-based model (more precisely a model where all embeddings
        must have the same dimensions). If `True`, the param `cat_embed_cols`
        must just be a list containing just the categorical column names:
        e.g.
        _['education', 'relationship', ...]_. This is because they will all be
         encoded using embeddings of the same dim, which will be specified
         later when the model is defined. <br/> Param alias:
         `for_transformer`
    with_cls_token: bool, default = False
        Boolean indicating if a `'[CLS]'` token will be added to the dataset
        when using attention-based models. The final hidden state
        corresponding to this token is used as the aggregated representation
        for classification and regression tasks. If not, the categorical
        (and continuous embeddings if present) will be concatenated before
        being passed to the final MLP (if present).
    shared_embed: bool, default = False
        Boolean indicating if the embeddings will be "shared" when using
        attention-based models. The idea behind `shared_embed` is
        described in the Appendix A in the [TabTransformer paper](https://arxiv.org/abs/2012.06678):
        _'The goal of having column embedding is to enable the model to
        distinguish the classes in one column from those in the other
        columns'_. In other words, the idea is to let the model learn which
        column is embedded at the time. See: `pytorch_widedeep.models.transformers._layers.SharedEmbeddings`.
    verbose: int, default = 1
    scale: bool, default = False
        :information_source: **note**: this arg will be removed in upcoming
         releases. Please use `cols_to_scale` instead. <br/> Bool indicating
         whether or not to scale/standarise continuous cols. It is important
         to emphasize that all the DL models for tabular data in the library
         also include the possibility of normalising the input continuous
         features via a `BatchNorm` or a `LayerNorm`. <br/> Param alias:
         `scale_cont_cols`.
    already_standard: List, default = None
        :information_source: **note**: this arg will be removed in upcoming
         releases. Please use `cols_to_scale` instead. <br/> List with the
         name of the continuous cols that do not need to be
         scaled/standarised.

    Other Parameters
    ----------------
    **kwargs: dict
        `pd.cut` and `StandardScaler` related args

    Attributes
    ----------
    cat_cols: List[str]
        List containing the names of the categorical columns after processing.
    cols_and_bins: Optional[Dict[str, Union[int, List[float]]]]
        Dictionary containing the quantization setup after processing if
        quantization is requested. This is derived from the
        quantization_setup parameter.
    quant_args: Dict
        Dictionary containing arguments passed to pandas.cut() function for quantization.
    scale_args: Dict
        Dictionary containing arguments passed to StandardScaler.
    label_encoder: LabelEncoder
        Instance of LabelEncoder used to encode categorical variables.
        See `pytorch_widedeep.utils.dense_utils.LabelEncoder`.
    cat_embed_input: List
        List of tuples with the column name, number of individual values for
        that column and, if `with_attention` is set to `False`, the
        corresponding embeddings dim, e.g. _[('education', 16, 10),
        ('relationship', 6, 8), ...]_.
    standardize_cols: List
        List of the columns that will be standardized.
    scaler: StandardScaler
        An instance of `sklearn.preprocessing.StandardScaler` if standardization
        is requested.
    column_idx: Dict
        Dictionary where keys are column names and values are column indexes.
        This is necessary to slice tensors.
    quantizer: Quantizer
        An instance of `Quantizer` if quantization is requested.
    is_fitted: bool
        Boolean indicating if the preprocessor has been fitted.

    Examples
    --------
    >>> import pandas as pd
    >>> import numpy as np
    >>> from pytorch_widedeep.preprocessing import ChunkTabPreprocessor
    >>> np.random.seed(42)
    >>> chunk_df = pd.DataFrame({'cat_col': np.random.choice(['A', 'B', 'C'], size=8),
    ... 'cont_col': np.random.uniform(1, 100, size=8)})
    >>> cat_embed_cols = [('cat_col',4)]
    >>> cont_cols = ['cont_col']
    >>> tab_preprocessor = ChunkTabPreprocessor(
    ... n_chunks=1, cat_embed_cols=cat_embed_cols, continuous_cols=cont_cols
    ... )
    >>> X_tab = tab_preprocessor.fit_transform(chunk_df)
    >>> tab_preprocessor.cat_embed_cols
    [('cat_col', 4)]
    >>> tab_preprocessor.column_idx
    {'cat_col': 0, 'cont_col': 1}
    """

    @alias("with_attention", ["for_transformer"])
    @alias("cat_embed_cols", ["embed_cols"])
    @alias("scale", ["scale_cont_cols"])
    @alias("cols_and_bins", ["quantization_setup"])
    def __init__(
        self,
        n_chunks: int,
        cat_embed_cols: Optional[Union[List[str], List[Tuple[str, int]]]] = None,
        continuous_cols: Optional[List[str]] = None,
        cols_and_bins: Optional[Dict[str, List[float]]] = None,
        cols_to_scale: Optional[Union[List[str], str]] = None,
        default_embed_dim: int = 16,
        with_attention: bool = False,
        with_cls_token: bool = False,
        shared_embed: bool = False,
        verbose: int = 1,
        *,
        scale: bool = False,
        already_standard: Optional[List[str]] = None,
        **kwargs,
    ):
        super(ChunkTabPreprocessor, self).__init__(
            cat_embed_cols=cat_embed_cols,
            continuous_cols=continuous_cols,
            quantization_setup=None,
            cols_to_scale=cols_to_scale,
            auto_embed_dim=False,
            embedding_rule="google",  # does not matter, irrelevant
            default_embed_dim=default_embed_dim,
            with_attention=with_attention,
            with_cls_token=with_cls_token,
            shared_embed=shared_embed,
            verbose=verbose,
            scale=scale,
            already_standard=already_standard,
            **kwargs,
        )

        self.n_chunks = n_chunks
        self.chunk_counter = 0

        self.cols_and_bins = cols_and_bins  # type: ignore[assignment]
        if self.cols_and_bins is not None:
            self.quantizer = Quantizer(self.cols_and_bins, **self.quant_args)

        self.embed_prepared = False
        self.continuous_prepared = False

    def partial_fit(self, df: pd.DataFrame) -> "ChunkTabPreprocessor":  # noqa: C901
        # df here, and throughout the class, is a chunk of the original df
        self.chunk_counter += 1

        df_adj = self._insert_cls_token(df) if self.with_cls_token else df.copy()

        self.column_idx: Dict[str, int] = {}

        # Categorical embeddings logic
        if self.cat_embed_cols is not None:
            if not self.embed_prepared:
                df_cat, self.cat_embed_dim = self._prepare_categorical(df_adj)
                self.label_encoder = LabelEncoder(
                    columns_to_encode=df_cat.columns.tolist(),
                    shared_embed=self.shared_embed,
                    with_attention=self.with_attention,
                )
                self.label_encoder.partial_fit(df_cat)
            else:
                df_cat = df_adj[self.cat_cols]
                self.label_encoder.partial_fit(df_cat)

            self.column_idx.update({k: v for v, k in enumerate(df_cat.columns)})

        # Continuous columns logic
        if self.continuous_cols is not None:
            if not self.continuous_prepared:
                df_cont, self.cont_embed_dim = self._prepare_continuous(df_adj)
            else:
                df_cont = df[self.continuous_cols]

            self.column_idx.update(
                {k: v + len(self.column_idx) for v, k in enumerate(df_cont.columns)}
            )

            if self.standardize_cols is not None:
                self.scaler.partial_fit(df_cont[self.standardize_cols].values)

        if self.chunk_counter == self.n_chunks:
            if self.cat_embed_cols is not None or self.cols_and_bins is not None:
                self.cat_embed_input: List[
                    Union[Tuple[str, int], Tuple[str, int, int]]
                ] = []

            if self.cat_embed_cols is not None:
                for k, v in self.label_encoder.encoding_dict.items():
                    if self.with_attention:
                        self.cat_embed_input.append((k, len(v)))
                    else:
                        self.cat_embed_input.append((k, len(v), self.cat_embed_dim[k]))

            if self.cols_and_bins is not None:
                assert self.cont_embed_dim is not None  # just to make mypy happy
                if self.with_attention:
                    for col, n_cat, _ in self.cont_embed_dim:
                        self.cat_embed_input.append((col, n_cat))
                else:
                    self.cat_embed_input.extend(self.cont_embed_dim)

            self.is_fitted = True

        return self

    def fit(self, df: pd.DataFrame) -> "ChunkTabPreprocessor":
        # just to override the fit method in the base class. This class is not
        # designed or thought to run fit
        return self.partial_fit(df)

    def _prepare_categorical(
        self, df: pd.DataFrame
    ) -> Tuple[pd.DataFrame, Dict[str, int]]:
        # When dealing with chunks we will NOT support the option of
        # automatically define embeddings as this implies going through the
        # entire dataset
        if isinstance(self.cat_embed_cols[0], tuple):
            self.cat_cols: List[str] = [emb[0] for emb in self.cat_embed_cols]
            cat_embed_dim: Dict[str, int] = dict(self.cat_embed_cols)  # type: ignore
        else:
            self.cat_cols = self.cat_embed_cols  # type: ignore[assignment]
            cat_embed_dim = {
                e: self.default_embed_dim for e in self.cat_embed_cols  # type: ignore[misc]
            }  # type: ignore

        self.embed_prepared = True

        return df[self.cat_cols], cat_embed_dim

    def _prepare_continuous(
        self, df: pd.DataFrame
    ) -> Tuple[pd.DataFrame, Optional[List[Tuple[str, int, int]]]]:
        # Standardization logic
        if self.cols_to_scale is not None:
            self.standardize_cols = (
                self.cols_to_scale
                if self.cols_to_scale != "all"
                else self.continuous_cols
            )
        elif self.scale:
            if self.already_standard is not None:
                self.standardize_cols = [
                    c
                    for c in self.continuous_cols  # type: ignore[misc]
                    if c not in self.already_standard
                ]
            else:
                self.standardize_cols = self.continuous_cols
        else:
            self.standardize_cols = None

        if self.standardize_cols is not None:
            self.scaler = StandardScaler(**self.scale_args)
        elif self.verbose:
            warnings.warn("Continuous columns will not be normalised")

        # Quantization logic
        if self.cols_and_bins is not None:
            # the quantized columns are then treated as categorical
            quant_cont_embed_input: Optional[List[Tuple[str, int, int]]] = []
            for col, val in self.cols_and_bins.items():
                if isinstance(val, int):
                    quant_cont_embed_input.append(
                        (
                            col,
                            val,
                            self.default_embed_dim,
                        )
                    )
                else:
                    quant_cont_embed_input.append(
                        (
                            col,
                            len(val) - 1,
                            self.default_embed_dim,
                        )
                    )
        else:
            quant_cont_embed_input = None

        self.continuous_prepared = True

        return df[self.continuous_cols], quant_cont_embed_input

    def __repr__(self) -> str:  # noqa: C901
        list_of_params: List[str] = []
        if self.n_chunks is not None:
            list_of_params.append("n_chunks={n_chunks}")
        if self.cat_embed_cols is not None:
            list_of_params.append("cat_embed_cols={cat_embed_cols}")
        if self.continuous_cols is not None:
            list_of_params.append("continuous_cols={continuous_cols}")
        if self.cols_and_bins is not None:
            list_of_params.append("cols_and_bins={cols_and_bins}")
        if self.cols_to_scale is not None:
            list_of_params.append("cols_to_scale={cols_to_scale}")
        if self.default_embed_dim != 16:
            list_of_params.append("default_embed_dim={default_embed_dim}")
        if self.with_attention:
            list_of_params.append("with_attention={with_attention}")
        if self.with_cls_token:
            list_of_params.append("with_cls_token={with_cls_token}")
        if self.shared_embed:
            list_of_params.append("shared_embed={shared_embed}")
        if self.verbose != 1:
            list_of_params.append("verbose={verbose}")
        if self.scale:
            list_of_params.append("scale={scale}")
        if self.already_standard is not None:
            list_of_params.append("already_standard={already_standard}")
        if len(self.quant_args) > 0:
            list_of_params.append(
                ", ".join([f"{k}" + "=" + f"{v}" for k, v in self.quant_args.items()])
            )
        if len(self.scale_args) > 0:
            list_of_params.append(
                ", ".join([f"{k}" + "=" + f"{v}" for k, v in self.scale_args.items()])
            )
        all_params = ", ".join(list_of_params)
        return f"ChunkTabPreprocessor({all_params.format(**self.__dict__)})"

ChunkTextPreprocessor

Bases: TextPreprocessor

Preprocessor to prepare the deeptext input dataset

Parameters:

Name Type Description Default
text_col str

column in the input dataframe containing either the texts or the filenames where the text documents are stored

 required
n_chunks int

Number of chunks that the text dataset is divided by.

 required
root_dir Optional[str]

If 'text_col' contains the filenames with the text documents, this is the path to the directory where those documents are stored.

 None
max_vocab int

Maximum number of tokens in the vocabulary

 30000
min_freq int

Minimum frequency for a token to be part of the vocabulary

 5
maxlen int

Maximum length of the tokenized sequences

 80
pad_first bool

Indicates whether the padding index will be added at the beginning or the end of the sequences

 True
pad_idx int

padding index. Fastai's Tokenizer leaves 0 for the 'unknown' token.

 1
word_vectors_path Optional[str]

Path to the pretrained word vectors

 None
n_cpus Optional[int]

number of CPUs to used during the tokenization process

 None
verbose int

Enable verbose output.

 1

Attributes:

Name Type Description
vocab Vocab

an instance of pytorch_widedeep.utils.fastai_transforms.ChunkVocab
embedding_matrix ndarray

Array with the pretrained embeddings if word_vectors_path is not None

Examples:

>>> import pandas as pd
>>> from pytorch_widedeep.preprocessing import ChunkTextPreprocessor
>>> chunk_df = pd.DataFrame({'text_column': ["life is like a box of chocolates",
... "You never know what you're gonna get"]})
>>> chunk_text_preprocessor = ChunkTextPreprocessor(text_col='text_column', n_chunks=1,
... max_vocab=25, min_freq=1, maxlen=10, verbose=0, n_cpus=1)
>>> processed_chunk = chunk_text_preprocessor.fit_transform(chunk_df)
Source code in pytorch_widedeep/preprocessing/text_preprocessor.py 
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
class ChunkTextPreprocessor(TextPreprocessor):
    r"""Preprocessor to prepare the ``deeptext`` input dataset

    Parameters
    ----------
    text_col: str
        column in the input dataframe containing either the texts or the
        filenames where the text documents are stored
    n_chunks: int
        Number of chunks that the text dataset is divided by.
    root_dir: str, Optional, default = None
        If 'text_col' contains the filenames with the text documents, this is
        the path to the directory where those documents are stored.
    max_vocab: int, default=30000
        Maximum number of tokens in the vocabulary
    min_freq: int, default=5
        Minimum frequency for a token to be part of the vocabulary
    maxlen: int, default=80
        Maximum length of the tokenized sequences
    pad_first: bool,  default = True
        Indicates whether the padding index will be added at the beginning or the
        end of the sequences
    pad_idx: int, default = 1
        padding index. Fastai's Tokenizer leaves 0 for the 'unknown' token.
    word_vectors_path: str, Optional
        Path to the pretrained word vectors
    n_cpus: int, Optional, default = None
        number of CPUs to used during the tokenization process
    verbose: int, default 1
        Enable verbose output.

    Attributes
    ----------
    vocab: Vocab
        an instance of `pytorch_widedeep.utils.fastai_transforms.ChunkVocab`
    embedding_matrix: np.ndarray
        Array with the pretrained embeddings if `word_vectors_path` is not None

    Examples
    ---------
    >>> import pandas as pd
    >>> from pytorch_widedeep.preprocessing import ChunkTextPreprocessor
    >>> chunk_df = pd.DataFrame({'text_column': ["life is like a box of chocolates",
    ... "You never know what you're gonna get"]})
    >>> chunk_text_preprocessor = ChunkTextPreprocessor(text_col='text_column', n_chunks=1,
    ... max_vocab=25, min_freq=1, maxlen=10, verbose=0, n_cpus=1)
    >>> processed_chunk = chunk_text_preprocessor.fit_transform(chunk_df)
    """

    def __init__(
        self,
        text_col: str,
        n_chunks: int,
        root_dir: Optional[str] = None,
        max_vocab: int = 30000,
        min_freq: int = 5,
        maxlen: int = 80,
        pad_first: bool = True,
        pad_idx: int = 1,
        already_processed: Optional[bool] = False,
        word_vectors_path: Optional[str] = None,
        n_cpus: Optional[int] = None,
        verbose: int = 1,
    ):
        super(ChunkTextPreprocessor, self).__init__(
            text_col=text_col,
            max_vocab=max_vocab,
            min_freq=min_freq,
            maxlen=maxlen,
            pad_first=pad_first,
            pad_idx=pad_idx,
            already_processed=already_processed,
            word_vectors_path=word_vectors_path,
            n_cpus=n_cpus,
            verbose=verbose,
        )

        self.n_chunks = n_chunks
        self.root_dir = root_dir

        self.chunk_counter = 0

        self.is_fitted = False

    def partial_fit(self, df: pd.DataFrame) -> "ChunkTextPreprocessor":
        # df is a chunk of the original dataframe
        self.chunk_counter += 1

        texts = self._read_texts(df, self.root_dir)

        tokens = get_texts(texts, self.already_processed, self.n_cpus)

        if not hasattr(self, "vocab"):
            self.vocab = ChunkVocab(
                max_vocab=self.max_vocab,
                min_freq=self.min_freq,
                pad_idx=self.pad_idx,
                n_chunks=self.n_chunks,
            )

        self.vocab.fit(tokens)

        if self.chunk_counter == self.n_chunks:
            if self.verbose:
                print("The vocabulary contains {} tokens".format(len(self.vocab.stoi)))
            if self.word_vectors_path is not None:
                self.embedding_matrix = build_embeddings_matrix(
                    self.vocab, self.word_vectors_path, self.min_freq
                )

            self.is_fitted = True

        return self

    def fit(self, df: pd.DataFrame) -> "ChunkTextPreprocessor":
        # df is a chunk of the original dataframe
        return self.partial_fit(df)

    def __repr__(self) -> str:
        list_of_params: List[str] = ["text_col='{text_col}'"]
        if self.n_chunks is not None:
            list_of_params.append("n_chunks={n_chunks}")
        if self.root_dir is not None:
            list_of_params.append("root_dir={root_dir}")
        list_of_params.append("max_vocab={max_vocab}")
        list_of_params.append("min_freq={min_freq}")
        list_of_params.append("maxlen={maxlen}")
        list_of_params.append("pad_first={pad_first}")
        list_of_params.append("pad_idx={pad_idx}")
        if self.word_vectors_path is not None:
            list_of_params.append("word_vectors_path={word_vectors_path}")
        if self.n_cpus is not None:
            list_of_params.append("n_cpus={n_cpus}")
        list_of_params.append("verbose={verbose}")
        all_params = ", ".join(list_of_params)
        return f"ChunkTextPreprocessor({all_params.format(**self.__dict__)})"

ChunkHFPreprocessor

Bases: HFPreprocessor

Text processor to prepare the deeptext input dataset that is a wrapper around HuggingFace's tokenizers.

Hugginface Tokenizer's are already 'trained'. Therefore, unlike the ChunkTextPreprocessor this is mostly identical to the HFPreprocessor with the only difference that the class needs a 'text_col' parameter to be passed. Also the parameter encode_params is not really optional when using this class. It must be passed containing at least the 'max_length' encoding parameter. This is because we need to ensure that all sequences have the same length when encoding in chunks.

Parameters:

Name Type Description Default
model_name str

The model name from the transformers library e.g. 'bert-base-uncased'. Currently supported models are those from the families: BERT, RoBERTa, DistilBERT, ALBERT and ELECTRA.

 required
text_col str

The column in the input dataframe containing the text data. When using the ChunkHFPreprocessor the text_col parameter is mandatory.

 required
root_dir Optional[str]

The root directory where the text files are located. This is only needed if the text data is stored in text files. If the text data is stored in a column in the input dataframe, this parameter is not needed.

 None
use_fast_tokenizer bool

Whether to use the fast tokenizer from HuggingFace or not

 True
num_workers Optional[int]

Number of workers to use when preprocessing the text data. If not None, and use_fast_tokenizer is False, the text data will be preprocessed in parallel using the number of workers specified. If use_fast_tokenizer is True, this argument is ignored.

 None
preprocessing_rules Optional[List[Callable[[str], str]]]

A list of functions to be applied to the text data before encoding. This can be useful to clean the text data before encoding. For example, removing html tags, special characters, etc.

 None
tokenizer_params Optional[Dict[str, Any]]

Additional parameters to be passed to the HuggingFace's PreTrainedTokenizer.

 None
encode_params Optional[Dict[str, Any]]

Additional parameters to be passed to the batch_encode_plus method of the HuggingFace's PreTrainedTokenizer. In the case of the ChunkHFPreprocessor, this parameter is not really Optional. It must be passed containing at least the 'max_length' encoding parameter

 None

Attributes:

Name Type Description
is_fitted bool

Boolean indicating if the preprocessor has been fitted. This is a HuggingFacea tokenizer, so it is always considered fitted and this attribute is manually set to True internally. This parameter exists for consistency with the rest of the library and because is needed for some functionality in the library.
Source code in pytorch_widedeep/preprocessing/hf_preprocessor.py 
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
class ChunkHFPreprocessor(HFPreprocessor):
    """Text processor to prepare the ``deeptext`` input dataset that is a
    wrapper around HuggingFace's tokenizers.

    Hugginface Tokenizer's are already 'trained'. Therefore, unlike the
    `ChunkTextPreprocessor` this is mostly identical to the `HFPreprocessor`
    with the only difference that the class needs a 'text_col' parameter to
    be passed. Also the parameter `encode_params` is not really optional when
    using this class. It must be passed containing at least the
    'max_length' encoding parameter. This is because we need to ensure that
     all sequences have the same length when encoding in chunks.

    Parameters
    ----------
    model_name: str
        The model name from the transformers library e.g. _'bert-base-uncased'_.
        Currently supported models are those from the families: BERT, RoBERTa,
        DistilBERT, ALBERT and ELECTRA.
    text_col: str, default = None
        The column in the input dataframe containing the text data. When using
        the `ChunkHFPreprocessor` the `text_col` parameter is mandatory.
    root_dir: Optional[str], default = None
        The root directory where the text files are located. This is only
        needed if the text data is stored in text files. If the text data is
        stored in a column in the input dataframe, this parameter is not
        needed.
    use_fast_tokenizer: bool, default = False
        Whether to use the fast tokenizer from HuggingFace or not
    num_workers: Optional[int], default = None
        Number of workers to use when preprocessing the text data. If not
        None, and `use_fast_tokenizer` is False, the text data will be
        preprocessed in parallel using the number of workers specified. If
        `use_fast_tokenizer` is True, this argument is ignored.
    preprocessing_rules: Optional[List[Callable[[str], str]]], default = None
        A list of functions to be applied to the text data before encoding.
        This can be useful to clean the text data before encoding. For
        example, removing html tags, special characters, etc.
    tokenizer_params: Optional[Dict[str, Any]], default = None
        Additional parameters to be passed to the HuggingFace's
        `PreTrainedTokenizer`.
    encode_params: Optional[Dict[str, Any]], default = None
        Additional parameters to be passed to the `batch_encode_plus` method
        of the HuggingFace's `PreTrainedTokenizer`. In the case of the
        `ChunkHFPreprocessor`, this parameter is not really `Optional`. It
        must be passed containing at least the 'max_length' encoding
        parameter

    Attributes
    ----------
    is_fitted: bool
        Boolean indicating if the preprocessor has been fitted. This is a
        HuggingFacea tokenizer, so it is always considered fitted and this
        attribute is manually set to True internally. This parameter exists
        for consistency with the rest of the library and because is needed
        for some functionality in the library.

    """

    def __init__(
        self,
        model_name: str,
        *,
        text_col: str,
        root_dir: Optional[str] = None,
        use_fast_tokenizer: bool = True,
        num_workers: Optional[int] = None,
        preprocessing_rules: Optional[List[Callable[[str], str]]] = None,
        tokenizer_params: Optional[Dict[str, Any]] = None,
        encode_params: Optional[Dict[str, Any]] = None,
    ):
        super().__init__(
            model_name=model_name,
            use_fast_tokenizer=use_fast_tokenizer,
            text_col=text_col,
            num_workers=num_workers,
            preprocessing_rules=preprocessing_rules,
            tokenizer_params=tokenizer_params,
            encode_params=encode_params,
        )

        self.root_dir = root_dir

        # when using in chunks encode_params is not really optional. I will
        # review types in due time
        if self.encode_params is None:
            raise ValueError(
                "The 'encode_params' dict must be passed to the ChunkHFTokenizer "
                "containing at least the 'max_length' encoding parameter"
            )

        if "padding" not in self.encode_params or not self.encode_params["padding"]:
            self.encode_params["padding"] = True

        if (
            "truncation" not in self.encode_params
            or not self.encode_params["truncation"]
        ):
            self.encode_params["truncation"] = True

    def __repr__(self):
        return (
            f"ChunkHFPreprocessor(text_col={self.text_col}, model_name={self.model_name}, "
            f"use_fast_tokenizer={self.use_fast_tokenizer}, num_workers={self.num_workers}, "
            f"preprocessing_rules={self.preprocessing_rules}, tokenizer_params={self.tokenizer_params}, "
            f"encode_params={self.encode_params}, root_dir={self.root_dir})"
        )