Dependencies updates jsoup 1.10.3 crawler-commons 0.8 Core Use ISO representation of time for modifiedtime in adaptivescheduler ...

StormCrawler 1.5 has just been released! It is an important road mark with the move to Elasticsearch 5.x and the implementation of long-a...

The next (and probably imminent) release of StormCrawler will contain an update of Elasticsearch to version 5.3. This is definitely a good...

Many websites rely on AJAX to provide smooth and reactive web applications and/or single page websites. While this works fine for humans u...

This tutorial explains how to configure Elasticsearch with StormCrawler.  We first bootstrap a StormCrawler project using the Maven a...

I will be running a full-day workshop on crawling with StormCrawler on the  24th April  in Berlin. See full details on   https://endoc...

How big did you say? I am often contacted by prospective clients to help them crawl the web on a very large scale or find questions such...