def __iter__(self):
self.tokenQueue = deque([])<EOL>while self.state():<EOL><INDENT>while self.stream.errors:<EOL><INDENT>yield {"<STR_LIT:type>": tokenTypes["<STR_LIT>"], "<STR_LIT:data>": self.stream.errors.pop(<NUM_LIT:0>)}<EOL><DEDENT>while self.tokenQueue:<EOL><INDENT>yield self.tokenQueue.popleft()<EOL><DEDENT><DEDENT>
This is where the magic happens.

        We do our usually processing through the states and when we have a token
        to return we yield the token which pauses processing until the next token
        is requested.