NLP datasets¶

This module contains the TextDataset class, which is the main dataset you should use for your NLP tasks. It automatically does the preprocessing steps described in text.transform. It also contains all the functions to quickly get a TextDataBunch ready.

Quickly assemble your data¶

You should get your data in one of the following formats to make the most of the fastai library and use one of the factory methods of one of the TextDataBunch classes:

• raw text files in folders train, valid, test in an ImageNet style,
• a csv (with no index or Header) where the first column(s) gives the label(s) and the folowwing one the associated text,
• tokens and labels arrays already saved,
• ids, vocabulary (correspondance id to word) and labels already saved.

If you are assembling the data for a language model, you should define your labels as always 0 to respect those formats. The first time you create a DataBunch with one of those functions, your data will be preprocessed automatically and saved, so that the next time you call it is almost instantaneous.

Below are the classes that help assembling the raw data in a DataBunch suitable for NLP.

Create a DataBunch suitable for language modeling: all the texts in the datasets are concatenated and the labels are ignored. Instead, the target is the next word in the sentence.

Create a DataBunch suitable for a text classifier: all the texts are grouped by length (with a bit of randomness for the training set) then padded.

Create a DataBunch with the raw texts. This is only going to work if they all ahve the same lengths.

Factory methods (TextDataBunch)¶

All those classes have the following factory methods.

This function will create a DataBunch from texts placed in path in a train, valid and maybe test folders. Text files in the train and valid folders should be places in subdirectories according to their classes (always the same for a language model) and the ones for the test folder should all be placed there directly. tokenizer will be used to parse those texts into tokens. The shuffle flag will optionally shuffle the texts found.

You can pass a specific vocab for the numericalization step (if you are building a classifier from a language model you fine-tuned for instance). kwargs will be split between the TextDataset function and to the class initialization, you can precise there parameters such as max_vocab, chunksize, min_freq, n_labels (see the TextDataset documentation) or bs, bptt and pad_idx (see the sections LM data and classifier data).

This function will create a DataBunch from texts placed in path in a train.csv, valid.csv and maybe test.csv files. These csv files should have no header or index, and the label(s) should be the first column(s) (be sure to adjust the parameter n_labels if you have more than one). tokenizer will be used to parse those texts into tokens.

You can pass a specific vocab for the numericalization step (if you are building a classifier from a language model you fine-tuned for instance). kwargs will be split between the TextDataset function and to the class initialization, you can precise there parameters such as max_vocab, chunksize, min_freq, n_labels (see the TextDataset documentation) or bs, bptt and pad_idx (see the sections LM data and classifier data).

This function will create a DataBunch from texts already tokenized placed in path in files named f{train}{tok_suff}.npy, f{train}{lbl_suff}.npy, f{valid}{tok_suff}.npy, f{valid}{lbl_suff}.npy and maybe f{test}{tok_suff}.npy. If no label file exists, labels will default to all zeros. tok_suff and lbl_suff are '_tok' and '_lbl' respectively.

You can pass a specific vocab for the numericalization step (if you are building a classifier from a language model you fine-tuned for instance). kwargs will be split between the TextDataset function and to the class initialization, you can precise there parameters such as max_vocab, chunksize, min_freq, n_labels, tok_suff and lbl_suff (see the TextDataset documentation) or bs, bptt and pad_idx (see the sections LM data and classifier data).

This function will create a DataBunch from texts already tokenized placed in path in files named f{train}{id_suff}.npy, f{train}{lbl_suff}.npy, f{valid}{id_suff}.npy, f{valid}{lbl_suff}.npy and maybe f{test}{id_suff}.npy. If no label file exists, labels will default to all zeros. id_suff and lbl_suff are '_ids' and '_lbl' respectively. The itos file should contain the correspondance from ids to words.

kwargs will be split between the TextDataset function and to the class initialization, you can precise there parameters such as max_vocab, chunksize, min_freq, n_labels, tok_suff and lbl_suff (see the TextDataset documentation) or bs, bptt and pad_idx (see the sections LM data and classifier data).

This function will create a DataBunch in path from texts already processed into trn_ids, trn_lbls, val_ids, val_lbls and maybe tst_ids. You can specify the corresponding classes if applciable. You must specify the vocab_size so that the RNNLearner class can later infer the corresponding sizes in the model it will create. kwargs will be passed to the class initialization.

Example¶

Untar the IMDB sample dataset if not already done:

Since it comes in the form of csv files, we will use the corresponding text_data method. Here is an overview of what your file you should look like:

And here is a simple way of creating your DataBunch for language modelling or classification.

The TextDataset class¶

Behind the scenes, the previous functions will create a training, validation and maybe test TextDataset which is the class responsible for collecting and preprocessing the data.

This class shouldn't be initialized directly as it will rely on internal files being put in an 'tmp' folder of path. tokenizer and vocab will be used to tokenize and numericalize the texts (if needed). max_vocab and min_freq are passed at the create of the vocabulary (if needed). chunksize is the size of chunks preprocessed when loading the data from csv or folders. name is the name of the set that will be used to name the temporary files. n_labels is the number of labels if creating the data from a csv file. classes is the correspondance between label and classe. create_mtd is an internal flag that tells the TextDataset how it was created. It can be:

• CSV if it was created from texts or csv
• TOK if it was created from tokens (which means the TextDataset will always skip the tokenization)
• IDS if it was created from tokens (which means the TextDataset will always skip the tokenization and the numericalization)

Factory methods (TextDataset)¶

Instead of using the TextDataset init method, one of the following factory functions should be used instead:

Creates a TextDataset named name by scanning the subfolders in folder and using tokenizer. If classes are passed, only the subfolders named accordingly are checked. If shuffle is True, the data will be shuffled. Any additional kwargs are passed to the init method of TextDataset.

Creates a TextDataset named name by scanning the text files in folder and using tokenizer. All files are labelled classes[0] so this is typically used for the test set. If shuffle is True, the data will be shuffled. Any additional kwargs are passed to the init method of TextDataset.

Creates a TextDataset named name from tokens and labels saved in f{name}{tok_suff}.npy and f{name}{lbl_suff}.npy respectively. Any additional kwargs are passed to the init method of TextDataset.

Creates a TextDataset named name from ids, labels and dictionary saved in f{name}{id_suff}.npy, f{name}{lbl_suff}.npy and itos respectively. Any additional kwargs are passed to the init method of TextDataset.

Preprocessing¶

The internal preprocessing is done by the two following methods:

Internally, the TextDataset will create a 'tmp' folder in which he will copy or save the following files:

• name.csv (if created from folders or csv)
• name_tok.npy and name_lbl.npy (created by TextDataset.tokenize from the last step or copied if created from tokens)
• name_ids.npy, name_lbl.npy and itos (created by TextDataset.numericalize from the last step or copied if created from ids)

Then, when you invoke the TextDataset again, it will look for those temporary files and check their consistency to use them, in order to avoid doing again the numericalization or the tokenization. If you feel those files have been corrupted in any way, the following method will clear the 'tmp' subfolder of those files:

Internal methods¶

Language Model data¶

A language model is trained to guess what the next word is inside a flow of words. We don't feed it the different texts separately but concatenate them all together in a big array. To create the batches, we split this array into bs chuncks of continuous texts. Note that in all NLP tasks, we use the pytoch convention of sequence length being the first dimension (and batch size being the second one) so we transpose that array so that we can read the chunks of texts in columns. Here is an example of batch from our imdb sample dataset.

Then, as suggested in this article from Stephen Merity et al., we don't use a fixed bptt through the different batches but slightly change it from batch to batch.

This is all done internally when we use TextLMDataBunch, by creating DataLoader using the following class:

Takes the texts from dataset and concatenate them all, then create a big array with bs columns (transposed from the data source so that we read the texts in the columns). Spits batches with a size approximately equal to bptt but changing at every batch. If backwards is True, reverses the original text.

Called at the inialization to create the big array of text ids from the data array.

Classifier data¶

When preparing the data for a classifier, we keep the different texts separate, which poses another challenge for the creation of batches: since they don't all have the same length, we can't easily collate them together in batches. To help with this we use two different techniques:

• padding: each text is padded with the PAD token to get all the ones we picked to the same size
• sorting the texts (ish): to avoid having together a very long text with a very short one (which would then have a lot of PAD tokens), we regroup the texts by order of length. For the training set, we still add some randomness to avoid showing the same batches at every step of the training.

Here is an example of batch with padding (the padding index is 1, and the padding is applied before the sentences start).

This is all done internally when we use TextClasDataBunch, by using the following classes:

pytorch Sampler to batchify the data_source by order of length of the texts. Used for the validation and (if applicable) the test set.

pytorch Sampler to batchify with size bs the data_source by order of length of the texts with a bit of randomness. Used for the training set.

Function used by the pytorch DataLoader to collate the samples in batches while adding padding with pad_idx. If pad_first is True, padding is applied at the beginning (before the sentence starts) otherwise it's applied at the end.

Enums¶

Undocumented Methods - Methods moved below this line will intentionally be hidden¶

New Methods - Please document or move to the undocumented section¶