Concatenative language
Concatenative topics
Concatenative meta
Other languages
Meta
- make filters compile somehow - random sleep - redirects - https - cookies - connect timeout, page timeout, data timeout, overall timeout, stopping spiders if overall timeout is reached - parse robots.txt and make filters for it - flag to disable robots.txt ;) Not immediately needed - parallel version - retry framework - retry connection-failed - option to turn off dns caching - proxies - option to check if pages exist but not download them - custom user agent string - custom http headers - spidering of results of a spider - save to database - save to directories/files - follow relative links only - support ftp spidering - bytes per second download rate limit - download quota - quiet mode - prefer ipv4/ipv6
Describe this revision:
Save