How does Storm integrate with queues and databases?
Last reviewed:
Storm connects to message queues through spouts and to databases through ordinary client code inside bolts. A spout reads from a queue such as Kafka, RabbitMQ, Kestrel or JMS and emits a stream of tuples. Bolts then read from or write to any database with its normal driver, while Storm handles the parallelism, partitioning and retries on failure.
Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Example queue integrations include:
(As of 2012. Modern Apache Storm releases ship maintained connectors as part of the main project, including the storm-kafka-client spout.)
Likewise, integrating Storm with database systems is easy. Simply open a connection to your database and read/write like you normally would. Storm will handle the parallelization, partitioning, and retrying on failures when necessary.