def tokenize(self, data_source, callback=None):
self._reinitialize()<EOL>if callback is not None:<EOL><INDENT>self._deliver = callback<EOL><DEDENT>while True:<EOL><INDENT>frame = data_source.read()<EOL>if frame is None:<EOL><INDENT>break<EOL><DEDENT>self._current_frame += <NUM_LIT:1><EOL>self._process(frame)<EOL><DEDENT>self._post_process()<EOL>if callback is None:<EOL><INDENT>_ret = self._tokens<EOL>self._tokens = None<EOL>return _ret<EOL><DEDENT>
Read data from `data_source`, one frame a time, and process the read frames in
order to detect sequences of frames that make up valid tokens.

:Parameters:
   `data_source` : instance of the :class:`DataSource` class that implements a `read` method.
       'read' should return a slice of signal, i.e. frame (of whatever \
       type as long as it can be processed by validator) and None if \
       there is no more signal.

   `callback` : an optional 3-argument function.
       If a `callback` function is given, it will be called each time a valid token
       is found.


:Returns:
   A list of tokens if `callback` is None. Each token is tuple with the following elements:

    .. code python

        (data, start, end)

   where `data` is a list of read frames, `start`: index of the first frame in the
   original data and `end` : index of the last frame.