A powerful C# web crawler that makes advanced crawling features easy to use. AbotX builds upon the open source Abot C# Web Crawler by providing a powerful set of wrappers and extensions.

Crawl multiple sites concurrently
Pause/resume live crawls
Render jаvascript before processing
Simplified pluggability/extensibility
Avoid getting blocked by sites
Automatically tune speed/concurrency

A powerful C# web crawler that makes advanced crawling features easy to use. AbotX builds upon the open source Abot C# Web Crawler by providing a powerful set of wrappers and extensions.

A powerful C# web crawler that makes advanced crawling features easy to use. AbotX builds upon Abot C# Web Crawler Framework by providing a powerful set of wrappers and extensions.

Features

Crawl multiple sites concurrently (ParallelCrawlerEngine)
Pause/resume live crawls (CrawlerX & ParallelCrawlerEngine)
Render jаvascript before processing (CrawlerX & ParallelCrawlerEngine)
Simplified pluggability/extensibility (CrawlerX & ParallelCrawlerEngine)
Avoid getting blocked by sites (AutoThrottling)
Automatically tune speed/concurrency (AutoTuning)

A powerful C# web crawler that makes advanced crawling features easy to use. AbotX builds upon the open source Abot C# Web Crawler by providing a powerful set of wrappers and extensions.

Crawl multiple sites concurrently
Pause/resume live crawls
Render jаvascript before processing
Simplified pluggability/extensibility
Avoid getting blocked by sites
Automatically tune speed/concurrency