Distributed and fault-tolerant realtime computation
Last reviewed:
Storm and Flink both process events one at a time with low latency. Flink builds managed state, exactly-once checkpoints and event-time processing into its core API, and adds SQL on top. Storm's core API gives at-least-once processing, and state and exactly-once updates come from extra layers such as Trident and stateful bolts. Flink covers more of what a stateful pipeline needs out of the box. Storm keeps a smaller model that is easy to follow for stateless routing and enrichment.
Storm was built at BackType and open-sourced by Twitter in 2011, as one of the first open source systems for realtime computation. Flink grew out of Stratosphere, a research project at TU Berlin. It entered the Apache Incubator in 2014 and became a top-level Apache project at the end of that year, the same year as Storm.
| Storm | Flink | |
|---|---|---|
| Processing model | One tuple at a time. Trident adds micro-batches. | One record at a time, for both streams and bounded (batch) data. |
| Delivery guarantee | At-least-once through tuple tracking and replay. Exactly-once state through Trident. | Exactly-once state through periodic checkpoints. End-to-end exactly-once with transactional sinks such as Kafka. |
| State | Kept in an external store from inside bolts, or in stateful bolts that checkpoint to stores such as Redis. | Managed by Flink, held in memory or in RocksDB on local disk, and included in every checkpoint. |
| Time and windows | Windowed bolts since Storm 1.0, with event-time support through timestamps and watermarks. | Event time and watermarks throughout the API. Tumbling, sliding and session windows. |
| APIs | Spouts and bolts, Trident, and the Streams API added in Storm 2.0. | DataStream API, Table API and SQL. |
| Upgrades | Kill the topology and resubmit. State that lives in an external store survives. | Take a savepoint, then restart the new version from it with state intact. |
| Cluster | Nimbus, Supervisors and ZooKeeper. | JobManager and TaskManagers, standalone or on YARN or Kubernetes. |
Flink injects markers called barriers into the stream at its sources. When an operator has received a barrier on every input, it writes a snapshot of its state and passes the barrier on. Once every operator has done so, Flink has a consistent checkpoint of the whole job. After a failure it rewinds the sources to the checkpoint's position and restores each operator's state, so every record affects the state exactly once. Records keep flowing between checkpoints, so there is no batch interval in the latency.
Storm takes a different route. The acker tracks each spout tuple through the topology and asks the spout to replay it if any step fails or times out. That guarantees nothing is lost, but a replayed tuple can update a counter twice. Trident fixes this by grouping tuples into batches with transaction ids and storing the id next to each value, so a replayed batch is recognised and skipped.
Storm is a reasonable choice for topologies that keep little or no state: parsing, filtering, routing events to the right queue, or calling a service for each record. Its model has fewer concepts to learn, and an existing Storm cluster that runs well does not need replacing for a new stateless topology. Bolts can also be written in non-JVM languages through the multilang protocol.
Flink is the stronger option when the job holds large state, such as per-user sessions, running aggregates over days, or joins between two streams. It also fits when results must follow event time even though events arrive late or out of order, and when analysts want to write the pipeline in SQL. Savepoints make it possible to change and redeploy a stateful job without losing what it has accumulated.