def _buffer_extract(self, data):
<EOL>entities = collections.deque(data.split(self._delimiter, -<NUM_LIT:1>))<EOL>if self._size_limit:<EOL><INDENT>if self.input_size + len(entities[<NUM_LIT:0>]) > self._size_limit:<EOL><INDENT>raise Exception('<STR_LIT>')<EOL><DEDENT>self._input_size += len(entities[<NUM_LIT:0>])<EOL><DEDENT>first_entry = entities.popleft()<EOL>if len(first_entry) > <NUM_LIT:0>:<EOL><INDENT>self._input.append(first_entry)<EOL><DEDENT>if len(entities) == <NUM_LIT:0>:<EOL><INDENT>return []<EOL><DEDENT>entities.appendleft('<STR_LIT>'.join(self._input))<EOL>self._input.clear()<EOL>self._input.append(entities.pop())<EOL>if self._size_limit:<EOL><INDENT>self._input_size = len(self._input[<NUM_LIT:0>])<EOL><DEDENT>return entities<EOL>
Extract takes an arbitrary string of input data and returns an array of
tokenized entities, provided there were any available to extract.  This
makes for easy processing of datagrams using a pattern like:

  tokenizer.extract(data).map { |entity| Decode(entity) }.each do ...