How to zoom out using esper?

Question

I looked at Esper (and Storm) to handle the stream. Esper seems to be doing exactly what I want ... i.e. roling tools, medians, complex queries, etc .... but I wonder, / p>

How can I scale to multiple instances using Esper?

As far as I understand, Storm handles distributed processing, but with Esper you yourself.

I did not need to do this for the future future, but as we grow, our data volumes would also have to scale. Most likely we will be hosted on Amazon EC2.

Do I need to run multiple servers and data with strings before sending them to my Esper app?

Is there a more graceful way to handle it?

-Sajal

Answer 1

You can run an instance of Esper in a bolt, which means that Storm will handle the tuple / event federation, and Esper will handle CEP on the events it receives in this bolt.

It has some code and information about embedding Esper in a storm bolt: http://tomdzk.wordpress.com/2011/09/28/storm-esper/

However ... You should have a precedent that supports relatively stateless Esper mechanisms that process a subset of the data.

For example: you calculate the average daily temperature in a city. If you do not distribute your tuples using shuffleGrouping based on the city field, then each Esper bolt may have a different dataset for each city.

In principle, be sure to read how data is distributed in the Storm topology before moving on to this architecture.

Answer 2

From your question, does EsperHA seem to matter? Have you looked at that?

EsperHA is a complete solution for the ESP / CEP event with zero downtime. It combines Esper with local in-memory caching, robust overflow to disk or database, and cluster configuration using hot standby capabilities.