def reduce_vocab(tokens, similarity=<NUM_LIT>, limit=<NUM_LIT:20>, sort_order=-<NUM_LIT:1>):
if <NUM_LIT:0> <= similarity <= <NUM_LIT:1>:<EOL><INDENT>similarity *= <NUM_LIT:100><EOL><DEDENT>if sort_order:<EOL><INDENT>tokens = set(tokens)<EOL>tokens_sorted = sorted(list(tokens), reverse=bool(sort_order < <NUM_LIT:0>))<EOL><DEDENT>else:<EOL><INDENT>tokens_sorted = list(tokens)<EOL>tokens = set(tokens)<EOL><DEDENT>thesaurus = {}<EOL>for tok in tokens_sorted:<EOL><INDENT>try:<EOL><INDENT>tokens.remove(tok)<EOL><DEDENT>except (KeyError, ValueError):<EOL><INDENT>continue<EOL><DEDENT>matches = fuzzy.extractBests(tok, list(tokens), score_cutoff=int(similarity), limit=limit)<EOL>if matches:<EOL><INDENT>thesaurus[tok] = list(zip(*matches))[<NUM_LIT:0>]<EOL><DEDENT>else:<EOL><INDENT>thesaurus[tok] = ()<EOL><DEDENT>for syn in thesaurus[tok]:<EOL><INDENT>tokens.discard(syn)<EOL><DEDENT><DEDENT>return thesaurus<EOL>
Find spelling variations of similar words within a list of tokens to reduce token set size

    Lexically sorted in reverse order (unless `reverse=False`), before running through fuzzy-wuzzy
    which results in the longer of identical spellings to be prefered (e.g. "ones" prefered to "one")
    as the key token. Usually you wantThis is usually what you want.

    Arguments:
      tokens (list or set or tuple of str): token strings from which to eliminate similar spellings
      similarity (float): portion of characters that should be unchanged in order to be considered a synonym
        as a fraction of the key token length.
        e.g. `0.85` (which means 85%) allows "hon" to match "on" and "honey", but not "one"

    Returns:
      dict: { 'token': ('similar_token', 'similar_token2', ...), ...}

    Examples:
      FIXME: can't doctest these because tuple order isn't consistent, must convert to sets
      >>> tokens = ('on', 'hon', 'honey', 'ones', 'one', 'two', 'three')
      >>> bool(reduce_vocab(tokens, sort_order=-1))  #  == answer)
      True

      >> answer = {'hon': ('on', 'honey'),
      ...           'one': ('ones',),
      ...           'three': (),
      ...           'two': ()}
      >> reduce_vocab(tokens, sort_order=1) == answer
      True
      >> answer = {'honey': ('hon',),
      ...           'ones': ('on', 'one'),
      ...           'three': (),
      ...           'two': ()}


      >> (reduce_vocab(tokens, similarity=0.3, limit=2, sort_order=-1) ==
      ...  {'ones': (), 'two': ('on', 'hon'), 'three': ('honey', 'one')})
      True
      >> (reduce_vocab(tokens, similarity=0.3, limit=3, sort_order=-1) ==
      ...  {'ones': (), 'two': ('on', 'hon', 'one'), 'three': ('honey',)})
      True