def parse(self, contents):
i = <NUM_LIT:0><EOL>for text in contents.split(self.delim):<EOL><INDENT>if not len(text.strip()):<EOL><INDENT>continue<EOL><DEDENT>words = text.split()<EOL>char_offsets = [<NUM_LIT:0>] + [<EOL>int(_) for _ in np.cumsum([len(x) + <NUM_LIT:1> for x in words])[:-<NUM_LIT:1>]<EOL>]<EOL>text = "<STR_LIT:U+0020>".join(words)<EOL>yield {<EOL>"<STR_LIT:text>": text,<EOL>"<STR_LIT>": words,<EOL>"<STR_LIT>": ["<STR_LIT>"] * len(words),<EOL>"<STR_LIT>": ["<STR_LIT>"] * len(words),<EOL>"<STR_LIT>": ["<STR_LIT>"] * len(words),<EOL>"<STR_LIT>": [<NUM_LIT:0>] * len(words),<EOL>"<STR_LIT>": ["<STR_LIT>"] * len(words),<EOL>"<STR_LIT>": char_offsets,<EOL>"<STR_LIT>": char_offsets,<EOL>}<EOL>i += <NUM_LIT:1><EOL><DEDENT>
Parse the document.

        :param contents: The text contents of the document.
        :rtype: a *generator* of tokenized text.