crawly

What's Changed

1. Created a simple management Web UI. Try it on localhost:4001
2. Added the possibility of creating spiders with the help of the YML format. Read more here: https://github.com/elixir-crawly/crawly/blob/master/documentation/spiders_in_yml.md
3. Added the possibility to run Crawly (and your scraping projects) without Elixir. Read more here: https://github.com/elixir-crawly/cr...

1. Bugfix for start_urls size (now it's possible to have very large start URLs)
2. Split business logs from other logs. Per spider logging
3. Send logs to CrawlyUI (optional)
4. Allow to override more spider options:
   • closespider_itemcount
   • closespider_timeout
   • concurrent_requests_per_domain (number of started workers)
5. Change on_spider_log_callback (now it also gets the crawl...

Release 0.12.0

Release 0.11.0

The release includes the following improvements:

1. WriteToFile pipeline now adds timestamps to filenames
2. WriteToFile pipeline will now create a folder if missing
3. SendToUI item pipeline will send data to experimental CrawlyUI management dashboard
4. Other smaller features