AbotX 2.1.3 Ultimate - A powerful C# web crawler A powerful C# web crawler that makes advanced crawling features easy to use. AbotX builds upon Abot C# Web Crawler Framework by providing a powerful set of wrappers and extensions.
Features
Crawl multiple sites concurrently (ParallelCrawlerEngine)
Pause/resume live crawls (CrawlerX & ParallelCrawlerEngine)
Render jаvascript before processing (CrawlerX & ParallelCrawlerEngine)
Simplified pluggability/extensibility (CrawlerX & ParallelCrawlerEngine)
Avoid getting blocked by sites (AutoThrottling)
Automatically tune speed/concurrency (AutoTuning)