Factor/To do/Spider
- retries
- redirects
- page timeout, data timeout, overall timeout, stopping spiders if overall timeout is reached
- https
- cookies
- make filters compile somehow
- parallel version
- custom user agent string
- spidering of results of a spider
- save to database
This revision created on Thu, 2 Oct 2008 01:10:16 by erg