def word_tokenize(text):
for (regexp, replacement) in RULES1:<EOL><INDENT>text = sub(regexp, replacement, text)<EOL><DEDENT>text = "<STR_LIT:U+0020>" + text + "<STR_LIT:U+0020>"<EOL>for (regexp, replacement) in RULES2:<EOL><INDENT>text = sub(regexp, replacement, text)<EOL><DEDENT>for regexp in CONTRACTIONS:<EOL><INDENT>text = sub(regexp, r"<STR_LIT>", text)<EOL><DEDENT>return text.split()<EOL>
Split string `text` into word tokens using the Penn Treebank rules