@staticmethod<EOL><INDENT>def get_tokens(tu, extent):<DEDENT>
tokens_memory = POINTER(Token)()<EOL>tokens_count = c_uint()<EOL>conf.lib.clang_tokenize(tu, extent, byref(tokens_memory),<EOL>byref(tokens_count))<EOL>count = int(tokens_count.value)<EOL>if count < <NUM_LIT:1>:<EOL><INDENT>return<EOL><DEDENT>tokens_array = cast(tokens_memory, POINTER(Token * count)).contents<EOL>token_group = TokenGroup(tu, tokens_memory, tokens_count)<EOL>for i in xrange(<NUM_LIT:0>, count):<EOL><INDENT>token = Token()<EOL>token.int_data = tokens_array[i].int_data<EOL>token.ptr_data = tokens_array[i].ptr_data<EOL>token._tu = tu<EOL>token._group = token_group<EOL>yield token<EOL><DEDENT>
Helper method to return all tokens in an extent.

        This functionality is needed multiple places in this module. We define
        it here because it seems like a logical place.