Distributed and fault-tolerant realtime computation
Last reviewed:
Storm processes each tuple the moment it arrives. Spark processes a stream as a series of small batches on the same engine it uses for batch jobs. Storm has no batch interval to wait for, so it suits pipelines where every event has to be handled straight away. Spark adds some latency but lets one codebase, and one cluster, cover both batch analytics and streaming.
A Storm job is a topology: spouts read from a source such as a queue, and bolts transform, join or store the tuples they receive. A topology runs until you kill it. When a bolt emits a tuple, Storm sends it to the next bolt immediately, and each spout and bolt has its own parallelism setting.
Spark added streaming in 2013 with Spark Streaming, which cuts a stream into small batches called DStreams. Spark 2.0 (2016) introduced Structured Streaming, which treats a stream as a table that keeps growing and lets you query it with the same DataFrame and SQL API used for batch data. The Spark documentation now calls DStreams a legacy API and points new work to Structured Streaming. Its guide puts micro-batch end-to-end latency at around 100 milliseconds at best, and around 1 millisecond for the continuous processing mode added in Spark 2.3, which drops to at-least-once guarantees.
| Storm | Spark (Structured Streaming) | |
|---|---|---|
| Processing model | One tuple at a time. Trident adds micro-batches. | Micro-batches by default. Continuous mode is experimental. |
| Delivery guarantee | At-least-once, with tuples replayed on failure. Exactly-once state through Trident. | Exactly-once end to end, given a replayable source and an idempotent or transactional sink. |
| API | Spouts and bolts. Storm 2.0 added a typed Streams API. | DataFrames, Datasets and SQL, shared with batch jobs. |
| Languages | Java and other JVM languages, plus any language through the multilang protocol. | Scala, Java, Python and SQL. |
| Cluster | Nimbus, Supervisors and ZooKeeper. | Standalone, YARN or Kubernetes. |
| Batch analytics | Not covered. Pair Storm with a batch system. | Same engine and code as streaming. |
Storm fits pipelines that act on each event on its own: routing, filtering, enrichment from a lookup, alerting. There is no batch boundary in the way, and the spout-and-bolt model makes it easy to see where a slow step sits and give it more executors. Teams with existing topologies also get a cluster that, once running, mostly restarts failed workers on its own. Bolts can also be written in languages outside the JVM.
Spark fits teams that already run Spark for batch work, or that want SQL over a stream. One job can join live events with a large historical table, and a model trained with Spark MLlib can score the same data. Exactly-once output comes with the core API rather than a separate layer. Managed Spark is available from Databricks, Amazon EMR, Google Dataproc and Azure, so a team does not have to run the cluster itself.
Storm was open-sourced in September 2011, about a year and a half before Spark Streaming appeared. Many early comparisons were written when Storm was the only widely used open source option for realtime processing. Twitter, which released Storm, announced in 2015 that it had replaced Storm internally with Heron, a new system that kept Storm's API. Storm itself continues as Apache Storm.