def __init__(self, texts_sample=None, jaccard_threshold=<NUM_LIT>, num_perm=<NUM_LIT>,<EOL>storage_config=None):
self.jaccard_threshold = jaccard_threshold<EOL>self.num_perm = num_perm<EOL>self.storage_config = storage_config<EOL>if storage_config:<EOL><INDENT>self.lsh = MinHashLSH(<EOL>threshold=self.jaccard_threshold, num_perm=self.num_perm,<EOL>storage_config=self.storage_config)<EOL><DEDENT>else:<EOL><INDENT>self.lsh = MinHashLSH(<EOL>threshold=self.jaccard_threshold, num_perm=self.num_perm)<EOL><DEDENT>self.too_common_shingles = set()<EOL>if texts_sample:<EOL><INDENT>self.too_common_shingles = get_too_common_shingles(texts_sample)<EOL><DEDENT>self.seen_urls = {}  <EOL>self.urls_by_path = defaultdict(set)  <EOL>self.urls_by_path_q = defaultdict(set)  <EOL>self.urls_by_path_qwp = defaultdict(set)  <EOL>self.params_by_path = defaultdict(set)  <EOL>self.param_values = defaultdict(set)  <EOL>self.path_dupstats = defaultdict(DupStat)<EOL>self.param_dupstats = defaultdict(DupStat)<EOL>self.path_param_dupstats = defaultdict(DupStat)<EOL>self.path_query_param_dupstats = defaultdict(DupStat)<EOL>self.param_value_dupstats = defaultdict(DupStat)<EOL>self.path_param_value_dupstats = defaultdict(DupStat)<EOL>
Initialize DupePredictor.
        :param jaccard_threshold: a minimal jaccard similarity when pages
        are considered duplicates (intersection of content / union of content).
        :param texts_sample: a list of texts to calculate too_common_shingles
        - this allows a more precise duplicate detection, because now
        we know which parts are common to all pages, and which are unique
        for each page.
        :param storage_config: configuration for a redis backend to persist
        minhashes in. Using this backend makes DupePredictor instances
        persistent across restarts. The configuration format is:
        storage_config={'type': 'redis', 'redis': {'host': 'localhost', 'port': 6379}}.
        See https://ekzhu.github.io/datasketch/lsh.html#minhash-lsh-at-scale