def word_tokenize(self, text, include_punc=True):
<EOL>if text.strip() == "<STR_LIT>":<EOL><INDENT>return []<EOL><DEDENT>_tokens = self.word_tok.tokenize(text)<EOL>if len(_tokens) == <NUM_LIT:1>:<EOL><INDENT>if _tokens[<NUM_LIT:0>] in PUNCTUATION:<EOL><INDENT>if include_punc:<EOL><INDENT>return _tokens<EOL><DEDENT>else:<EOL><INDENT>return []<EOL><DEDENT><DEDENT><DEDENT>if include_punc:<EOL><INDENT>return _tokens<EOL><DEDENT>else:<EOL><INDENT>words = [<EOL>word if word.startswith("<STR_LIT:'>") else strip_punc(<EOL>word,<EOL>all=False) for word in _tokens if strip_punc(<EOL>word,<EOL>all=False)]<EOL>return list(words)<EOL><DEDENT>
The Treebank tokenizer uses regular expressions to tokenize text as
        in Penn Treebank.

        It assumes that the text has already been segmented into sentences,
        e.g. using ``self.sent_tokenize()``.

        This tokenizer performs the following steps:

        - split standard contractions, e.g. ``don't`` -> ``do n't`` and ``they'll`` -> ``they 'll``
        - treat most punctuation characters as separate tokens
        - split off commas and single quotes, when followed by whitespace
        - separate periods that appear at the end of line

        Source: NLTK's docstring of ``TreebankWordTokenizer`` (accessed: 02/10/2014)