spider.py 0.5

Multithreaded crawling, reporting, and mirroring for Web and FTP

This module provides multithreaded crawling, reporting, and mirroring for Web

and FTP in one convenient library. Crawling depth, maximum number of URLs to

crawl, and maximum number of threads are user-configurable. Reports can be

generated on external URLS, internal redirects to outside URLs, unparsable HTML,

non-HTTP/FTP URLs, and broken links.