Storm

Distributed and fault-tolerant realtime computation

Storm 0.9.0 Released

8 Dec 2013

We are pleased to announce that Storm 0.9.0 has been released and is available from the downloads page. This release represents an important milestone in the evolution of Storm.

While a number of new features have been added, a key focus area for this release has been stability-related fixes. Though many users are successfully running work-in-progress builds for Storm 0.9.x in production, this release represents the most stable version to-date, and is highly recommended for everyone, especially users of 0.8.x versions.

Netty Transport

The first hightlight of this release is the new Netty Transport contributed by Yahoo! Engineering. Storm's core network transport mechanism is now plugable, and Storm now comes with two implementations: The original 0MQ transport, and a new Netty-based implementation.

In earlier versions, Storm relied solely on 0MQ for transport. Since 0MQ is a native library, it was highly platform-dependent and, at times, challenging to install properly. In addition, stability between versions varied widely between versions and only a relatively old 0MQ version (2.1.7) was certified to work with Storm.

The Netty transport offers a pure Java alternative that eliminates Storm's dependency on native libraries. The Netty transport's performance is up to twice as fast as 0MQ, and it will open the door for authorization and authentication between worker processes. For an in-depth performance comparison of the 0MQ and Netty transports, see this blog post by Storm contributor Bobby Evans.

To configure Storm to use the Netty transport simply add the following to your storm.yaml configuration and adjust the values to best suit your use-case:

storm.messaging.transport: "backtype.storm.messaging.netty.Context"
storm.messaging.netty.server_worker_threads: 1
storm.messaging.netty.client_worker_threads: 1
storm.messaging.netty.buffer_size: 5242880
storm.messaging.netty.max_retries: 100
storm.messaging.netty.max_wait_ms: 1000
storm.messaging.netty.min_wait_ms: 100

You can also write your own transport implementation by implementing the backtype.storm.messaging.IContext interface.

Log Viewer UI

Storm now includes a helpful new feature for debugging and monitoring topologies: The logviewer daemon.

In earlier versions of Storm, viewing worker logs involved determining a worker's location (host/port), typically through Storm UI, then sshing to that host and tailing the corresponding worker log file. With the new log viewer. You can now easily access a specific worker's log in a web browser by clicking on a worker's port number right from Storm UI.

The logviewer daemon runs as a separate process on Storm supervisor nodes. To enable the logviewer run the following command (under supervision) on your cluster's supervisor nodes:

$ storm logviewer

Improved Windows Support

In previous versions, running Storm on Microsoft Windows required installing third-party binaries (0MQ), modifying Storm's source, and adding Windows-specific scripts. With the addition of the platform-independent Netty transport, as well as numerous enhancements to make Storm more platform-independent, running Storm on Windows is easier than ever.

Security Improvements

Security, Authentication, and Authorization have been and will continue to be important focus areas for upcoming features. Storm 0.9.0 introduces an API for pluggable tuple serialization and a blowfish encryption based implementation for encrypting tuple data for sensitive use cases.

API Compatibility and Upgrading

For most Storm topology developers, upgrading to 0.9.0 is simply a matter of updating the dependency. Storm's core API has changed very little since the 0.8.2 release.

On the devops side, when upgrading to a new Storm release, it is safest to clear any existing state (Zookeeper, storm.local.dir), prior to upgrading.

Logging Changes

Another important change in 0.9.0 has to do with logging. Storm has largely switched over to the slf4j API (backed by a logback logger implementation). Some Storm dependencies rely on the log4j API, so Storm currently depends on log4j-over-slf4j.

These changes have implications for existing topologies and topology components that use the log4j API.

In general, and when possible, Storm topologies and topology components should use the slf4j API for logging.

Thanks

Special thanks are due to all those who have contributed to Storm -- whether through direct code contributions, documentation, bug reports, or helping other users on the mailing lists. Your efforts are much appreciated.

Changelog