def _fetch(self):
while True:<EOL><INDENT>url = self._to_fetch.get()<EOL>try:<EOL><INDENT>if url:<EOL><INDENT>page = self.get_page(url)<EOL>if page is None:    <EOL><INDENT>continue<EOL><DEDENT>for link, rel in page.links:<EOL><INDENT>if link not in self._seen:<EOL><INDENT>self._seen.add(link)<EOL>if (not self._process_download(link) and<EOL>self._should_queue(link, url, rel)):<EOL><INDENT>logger.debug('<STR_LIT>', link, url)<EOL>self._to_fetch.put(link)<EOL><DEDENT><DEDENT><DEDENT><DEDENT><DEDENT>finally:<EOL><INDENT>self._to_fetch.task_done()<EOL><DEDENT>if not url:<EOL><INDENT>break<EOL><DEDENT><DEDENT>
Get a URL to fetch from the work queue, get the HTML page, examine its
links for download candidates and candidates for further scraping.

This is a handy method to run in a thread.