Setting the cat amongst the pigeons

UPDATE: we have posted updated performance data. See this blog entry.

Ok.... so we're doing it again. We have done some benchmarking, and once again we've come ahead of the competition. This time its ESB performance.

A few weeks ago, we published some data on how the WSO2 ESB compares to a proprietary ESB. Because of what are known as Freedom of Use restrictions, we can't name that ESB - their license prohibits publishing any benchmark data.

But of course, with Open Source ESBs there aren't any such restrictions, so we have now benchmarked against Mule and Apache ServiceMix as well.

We chose three scenarios, all XML based, where we did the following things:

• Virtualization (routing)
• Content-Based Routing with an XPath
• XSLT-based transformation

The results were very interesting. Firstly, we had some significant issues with getting Mule to do these scenarios. When we did, the results were disappointing - for Mule. We came around 4x faster. Now, I'm hoping the Mule guys will get their act together and provide a decent contest, because we couldn't get Mule to do HTTP KeepAlive without failing, and also they keep sending unnecessary HEAD requests. You can read more about the Mule comparison on Asankha's blog.

ServiceMix did better. On one test - XSLT - they beat us. Of course we didn't take it sitting down - we did a fix to Apache Synapse and WSO2 ESB that gave us a 2x improvement on XSLT and now we can do 1800tps including transformation.

We also found a significant ServiceMix bug that means you couldn't use it in a real production environment with HTTP - if the number of incoming connections exceeds the thread pool then it crashes and you have to restart it. Would make for a nice DoS attack I guess. This is one area we've worked really hard on and we've tested our non-blocking transport with more than 2000 concurrent connections.

Anyway, enjoy. I'm expecting some fireworks!!