def tokenize(self):
self.tokens = []<EOL>self.terms = OrderedDict()<EOL>for token in utils.tokenize(self.text):<EOL><INDENT>if token['<STR_LIT>'] in self.stopwords:<EOL><INDENT>self.tokens.append(None)<EOL><DEDENT>else:<EOL><INDENT>self.tokens.append(token)<EOL>offsets = self.terms.setdefault(token['<STR_LIT>'], [])<EOL>offsets.append(token['<STR_LIT>'])<EOL><DEDENT><DEDENT>
Tokenize the text.