def lex(self, text):
pos = <NUM_LIT:0><EOL>while text:<EOL><INDENT>token_class, match = self.tokens.get_token(text)<EOL>if token_class is not None:<EOL><INDENT>matched_text = text[match.start():match.end()]<EOL>yield token_class(matched_text)<EOL>text = text[match.end():]<EOL>pos += match.end()<EOL><DEDENT>elif text[<NUM_LIT:0>] in self.blank_chars:<EOL><INDENT>text = text[<NUM_LIT:1>:]<EOL>pos += <NUM_LIT:1><EOL><DEDENT>else:<EOL><INDENT>raise LexerError(<EOL>'<STR_LIT>' % (text[<NUM_LIT:0>], text),<EOL>position=pos)<EOL><DEDENT><DEDENT>yield self.end_token()<EOL>
Split self.text into a list of tokens.

        Args:
            text (str): text to parse

        Yields:
            Token: the tokens generated from the given text.