Advanced Open Source Collage

Using Esper, Quartz, and Ganglia with Synapse

Here is the scenario. You have a bunch of machines that you want to monitor, and you want to kick off some process when the average CPU utilization stays at more than 90% for 4 hours - you are worried the machine might fry.

I recently built a similar scenario using 4 key open source components:

• Apache Synapse which is the ESB or glue that holds it together
• Ganglia which is the hardware monitoring toolkit
• Esper which can monitor the events and look for patterns
• Quartz which is a job scheduler

I've tried to capture this in a picture

So how does it work?
Lets start with GMond. Gmond is a neat little process from Ganglia that runs and captures information. It can send it via XDR/Multicast or XML/TCP. Rather than create a new transport for Synapse to accept incoming Ganglia XML, it seemed easier to me to poll for it. Basically if you open up a TCP socket to gmond it will pipe you the state of the machine as an XML statement.

This is where Quartz comes in. We've just added Quartz support to Synapse (we've been meaning to for ages). Quartz lets you run a job every n milliseconds, or based on a calendar, or all sorts of other settings.

The Quartz plugin for Synapse is really simple. You just have to drop the right JARs into the Synapse lib directory, and then add your Quartz XML config into the right place in the synapse.xml:
<startup>
<quartz:quartz >
<quartz:job>
<quartz:job-detail>
....
</quartz:quartz>
</startup>

So I wrote a really simple job that opens a socket, grabs the XML and "injects" this into Synapse. It implements the Quartz Job interface (basically execute()).
The code for the GMondPoller is here.

When you inject a message, you need to address it. Everything in Synapse is addressed by a URI, so I simply created a new "virtual" URI to indicate this is Ganglia XML - urn:gmond.

I then configured any message targeted for "urn:gmond" to be picked up by Esper. Last week I wrote an EsperMediator for Synapse, so that was already useable. I did a couple of improvements to that at the same time. The latest version is here.

Using the Esper configuration model, I told Esper about the Ganglia XML format, and I then set up an EQL statement to grab the average CPU USER across a batch of 100 messages. Of course in real life this could have been the 4 hourly sliding average for the scenario above.

Here is the Synapse config for Esper:
<class name="org.fremantle.esper.EsperMediator">
<property name="Configuration">
<esper-configuration xmlns="">
<event-type alias="Ganglia">
<xml-dom root-element-name="GANGLIA_XML">
<xpath-property property-name="cpu_user"
xpath="//GANGLIA_XML/CLUSTER/HOST/METRIC[@NAME='cpu_user']/@VAL" type="number"/>
</xml-dom>
</event-type>
</esper-configuration>
</property>
<property name="statement"
value="select avg(cpu_user) from Ganglia.win:length_batch(100) "/>
<property name="EventToAddress" value="urn:cpu"/>
</class>

After that, there wasn't much else to do. It would have been simple to send the resulting messages off to a JMS queue or other endpoint, but for the demo it was easier just to log them.

You can see the complete synapse.xml configuration here.

This is a great scenario. For me it shows some key SOA benefits:

• The fact that Ganglia and Esper both support XML meant it was simple to use Synapse to glue them together.
  
• It also meant the GMond poller job was really easy to write.
  
• Using virtual URIs as routing points within Synapse is a really nice model - its lightweight, simple and easy to understand
• I also really appreciate how flexible Esper is. It really exemplifies loose-coupling. With minor changes to the Esper statements and config, you could use this to fire an event based on almost any possible condition happening on the machine.
• I also like Quartz a lot, though I'm tempted to define a slightly simpler config model. The Quartz Java properties is nice, but the XML is a bit heavyweight, and I think the model we use in Synapse where we inject properties is a bit easier.

Anyway, I hope you enjoy the combination, and my thanks to Dave C. for suggesting it.