New
Release 0.13.0
- Bugfix for start_urls size (now it's possible to have very large start URLs)
- Split business logs from other logs. Per spider logging
- Send logs to CrawlyUI (optional)
- Allow to override more spider options:
- closespider_itemcount
- closespider_timeout
- concurrent_requests_per_domain (number of started workers)
- Change on_spider_log_callback (now it also gets the crawl_id)
- Parse pipelines