Distributed and fault-tolerant realtime computation
Last reviewed:
Kafka stores streams of records in a durable, partitioned log. Storm runs computation on streams. The two usually work together: Kafka holds the events, and a Storm topology reads them, processes them and writes the results somewhere else. People comparing the two often mean Storm and Kafka Streams, the stream processing library that has shipped with Kafka since version 0.10 in 2016.
Kafka was built at LinkedIn and open-sourced in 2011. Producers append records to topics. Each topic is split into partitions that are copied across several brokers, so the data survives a broker failure. Consumers read at their own pace and track their position with an offset, which means a consumer can stop, restart or rewind without losing data. Kafka keeps records for a set period whether or not anyone has read them.
Storm runs topologies of spouts and bolts across a cluster. It has no storage of its own. It needs a source, which is usually a queue, and it writes results to a database, another queue or a service. Storm handles the parallelism, sends tuples between machines and restarts work that fails.
The storm-kafka-client module provides a KafkaSpout. It reads from one or more topics, emits each record as a tuple, and commits the offset back to Kafka once the topology has fully processed that tuple. If a tuple fails, the spout reads it again from Kafka. That is how Storm gives at-least-once processing with Kafka as the source, and it relies on Kafka keeping records after they are read. A matching KafkaBolt writes tuples back to a topic.
Kafka Streams is a Java library you add to an application. There is no processing cluster to run: you start more copies of the application, and Kafka spreads the topic partitions among them.
| Storm | Kafka Streams | |
|---|---|---|
| Deployment | A Storm cluster (Nimbus, Supervisors, ZooKeeper) that runs submitted topologies. | A library inside your own application, deployed however you deploy that application. |
| Sources and sinks | Any source with a spout, including Kafka, RabbitMQ, JMS and files. Any sink from a bolt. | Kafka topics in and out. Kafka Connect moves data to and from other systems. |
| Scaling | Parallelism set per spout and bolt, changed with storm rebalance. | Limited by the number of input partitions. Add instances up to that number. |
| Delivery guarantee | At-least-once. Exactly-once state through Trident. | At-least-once by default. Exactly-once within Kafka since 0.11 (2017), using Kafka transactions. |
| State | External stores, Trident state or stateful bolts. | Local RocksDB stores, each backed by a Kafka changelog topic for recovery. |
| Languages | JVM languages, plus others through multilang. | Java and other JVM languages. |
If the data already lives in Kafka and the results go back to Kafka, Kafka Streams needs the least infrastructure: no separate cluster, and exactly-once with a single setting. Storm makes more sense when a pipeline reads from several kinds of sources, writes to systems other than Kafka, or needs different parallelism for each step. Storm still uses ZooKeeper to coordinate its cluster. Kafka removed its own ZooKeeper dependency in Kafka 4.0 (2025), so a new Kafka-only setup no longer needs a ZooKeeper ensemble at all.