def wiki(wiki_root, wiki_date, wiki_language, max_vocab_size=None):
data = WikiDumpStream(<EOL>root=os.path.expanduser(wiki_root), language=wiki_language,<EOL>date=wiki_date)<EOL>vocab = data.vocab<EOL>if max_vocab_size:<EOL><INDENT>for token in vocab.idx_to_token[max_vocab_size:]:<EOL><INDENT>vocab.token_to_idx.pop(token)<EOL><DEDENT>vocab.idx_to_token = vocab.idx_to_token[:max_vocab_size]<EOL><DEDENT>idx_to_counts = data.idx_to_counts<EOL>def code(shard):<EOL><INDENT>return [[vocab[token] for token in sentence if token in vocab]<EOL>for sentence in shard]<EOL><DEDENT>data = data.transform(code)<EOL>return data, vocab, idx_to_counts<EOL>
Wikipedia dump helper.

    Parameters
    ----------
    wiki_root : str
        Parameter for WikiDumpStream
    wiki_date : str
        Parameter for WikiDumpStream
    wiki_language : str
        Parameter for WikiDumpStream
    max_vocab_size : int, optional
        Specifies a maximum size for the vocabulary.

    Returns
    -------
    gluonnlp.data.DataStream
        Each sample is a valid input to
        gluonnlp.data.EmbeddingCenterContextBatchify.
    gluonnlp.Vocab
        Vocabulary of all tokens in the Wikipedia corpus as provided by
        WikiDumpStream but with maximum size max_vocab_size.
    idx_to_counts : list of int
        Mapping from token indices to their occurrence-counts in the Wikipedia
        corpus.