def preprocess_dataset(data, min_freq=<NUM_LIT:5>, max_vocab_size=None):
with print_time('<STR_LIT>'):<EOL><INDENT>counter = nlp.data.count_tokens(itertools.chain.from_iterable(data))<EOL>vocab = nlp.Vocab(counter, unknown_token=None, padding_token=None,<EOL>bos_token=None, eos_token=None, min_freq=min_freq,<EOL>max_size=max_vocab_size)<EOL>idx_to_counts = [counter[w] for w in vocab.idx_to_token]<EOL><DEDENT>def code(sentence):<EOL><INDENT>return [vocab[token] for token in sentence if token in vocab]<EOL><DEDENT>with print_time('<STR_LIT>'):<EOL><INDENT>data = data.transform(code, lazy=False)<EOL><DEDENT>data = nlp.data.SimpleDataStream([data])<EOL>return data, vocab, idx_to_counts<EOL>
Dataset preprocessing helper.

    Parameters
    ----------
    data : mx.data.Dataset
        Input Dataset. For example gluonnlp.data.Text8 or gluonnlp.data.Fil9
    min_freq : int, default 5
        Minimum token frequency for a token to be included in the vocabulary
        and returned DataStream.
    max_vocab_size : int, optional
        Specifies a maximum size for the vocabulary.

    Returns
    -------
    gluonnlp.data.DataStream
        Each sample is a valid input to
        gluonnlp.data.EmbeddingCenterContextBatchify.
    gluonnlp.Vocab
        Vocabulary of all tokens in Text8 that occur at least min_freq times of
        maximum size max_vocab_size.
    idx_to_counts : list of int
        Mapping from token indices to their occurrence-counts in the Text8
        dataset.