Grafana StormCrawler metrics v4

The Grafana dashboard for StormCrawler is a good starting point for monitoring the behaviour of your StormCrawler topology. This is typically used with Elasticsearch as a storage backend for the metrics generated by Storm but should work with any other Storm-compatible backend like Grafite or CloudWatch. 

Some of the metrics are specific to the components from the Elasticsearch module (spout, status, indexer) but you can simply remove or modify them if you use e.g. SOLR (NOTE: there was a feature request in Grafana to add SOLR as a datasource but to my knowledge, this is not yet available).

The latest version (4) brings the following changes.

• URLs waiting in queues 

The recent 1.8 release of StormCrawler added a new metrics for the FetcherBolt which allows tracking the amount of time URLs spend in the internal queues. This has been added to the "URLs waiting in queues" panel alongside the average population of the queues.

Average time spent in queues + average queues population

• ES StatusUpdater

Instead of tracking the number of bulk requests sent in the last minute, we now have a panel showing the evolution over time. This information is for the ES StatusUpdaterBolt only.

ES status updater bulk requests

• Acked in StatusBolt

This is a brand new panel which is not specific to Elasticsearch but operates on any component with 'status' for id and shows the number of tuples acked over time, broken down by source.  

Tuples acked by StatusUpdater

In the graph above, we can see a peak early in the crawl where most of the tuples acked came from the sitemap bolt. Please note that the values are stacked in this graph. Sitemap files are typically discovered early in a crawl and generate a large number of discovered URLs; this is not the case later on when most tuples come from the HTML parser.

• Robots panel

We removed the robots panel as the number of HTTP requests to robots files is shown in the "Fetcher: pages fetched" panel anyway and after the initial few minutes of a crawl, the panel simply indicated that the robots files were mostly cached.

• ES Indexed 

This is a new panel showing the number of documents indexed into Elasticsearch as well as the documents filtered out during the indexing.

What's new in StormCrawler 1.8

I have just released StormCrawler 1.8. As usual, here is a summary of the main changes:


Dependency updates

• Storm 1.2.1 #531
• SOLR 7.2.1 #528
• Tika 1.17 #518
• Elasticsearch 6.2.2 #525 and #539

Core

• Add option to send only N bytes of text to indexers #476
• BasicURLNormalizer to optionally convert IDN host names to ASCII/Punycode #522
• MemorySpout to generate tuples with DISCOVERED status #529
• OKHttp configure type of proxy #530
• http.content.limit inconsistent default to -1 #534
• Track time spent in the FetcherBolt queues #535
• Increase detect.charset.maxlength default value #537
• FeedParserBolt: metadata added by parse filters not passed forward in topology #541
• Use UTF-8 for input encoding of seeds (FileSpout) #542
• Default URL filter: exclude localhost and private address spaces #543
• URLStreamGrouping returns the taskIDs and not their index #547

WARC

• Upgrade WARC module to 1.1.0 version of storm-hdfs, fixes #520

SOLR

• Schema for status index needs date type for nextFetchDate #544
• SOLR indexer: use field type text for content field #545

Elasticsearch

• AggregationSpout fails with default value of es.status.bucket.field == _routing #521
• Move to Elasticsearch RESTAPi #539

We recommend all users to move to this version as it fixes several bugs (#541, #547) and adds some great new features. In particular, the use of the REST API for Elasticsearch, which makes the module future-proof but also easier to configure, but also #535 and #543.

As usual, thanks to all contributors and users. Happy crawling!