def tokenize(string):
for match in TOKENS_REGEX.finditer(string):<EOL><INDENT>yield Token(match.lastgroup, match.group().strip(), match.span())<EOL><DEDENT>
Match and yield all the tokens of the input string.