The crawler-conf.yaml generated by the archetypes leaves Storm's inter-executor batching at its default (topology.producer.batch.size: 1), so no batching happens between the spout, fetcher, parser, indexer and status updater executors.
Storm 3.0 adds topology.producer.batch.dynamic: when enabled together with a topology.producer.batch.size greater than 1, the effective batch size adapts at runtime between 1 and the configured size with AIMD, staying small under light load to keep latency low and growing under sustained load to preserve throughput. That removes the usual objection to batching in a crawl topology, where traffic is bursty and latency to the status updater matters.
Proposal: enable it in the archetype configurations with a conservative batch size below topology.max.spout.pending, and document the Storm topology settings that affect a crawl (topology.message.timeout.secs, topology.max.spout.pending, the producer batch settings and the flush interval), which are currently not covered in the configuration documentation.
The
crawler-conf.yamlgenerated by the archetypes leaves Storm's inter-executor batching at its default (topology.producer.batch.size: 1), so no batching happens between the spout, fetcher, parser, indexer and status updater executors.Storm 3.0 adds
topology.producer.batch.dynamic: when enabled together with atopology.producer.batch.sizegreater than 1, the effective batch size adapts at runtime between 1 and the configured size with AIMD, staying small under light load to keep latency low and growing under sustained load to preserve throughput. That removes the usual objection to batching in a crawl topology, where traffic is bursty and latency to the status updater matters.Proposal: enable it in the archetype configurations with a conservative batch size below
topology.max.spout.pending, and document the Storm topology settings that affect a crawl (topology.message.timeout.secs,topology.max.spout.pending, the producer batch settings and the flush interval), which are currently not covered in the configuration documentation.