Every cloud has a silver lining

WSO2 just published an update to our ESB performance testing.

If you recall, we published two sets of ESB performance testing last year. At the time, this was the only example of a openly published ESB benchmark. Since then, everyone has updated their ESBs, so it seemed like time to redo the benchmarking.

You can read the details here, and I suggest you do, because there are always interesting details that any summary cannot capture.

Before I talk about the results, I think its interesting that in a year, no-one else has published an open benchmark. So while any vendor sponsored benchmark is likely to be suspect, the fact is that we are the only people willing to openly benchmark ESBs and publish the code, and the results - whatever they are.

The results? Well, we still maintain a healthy lead on the other Open Source ESBs we tested. Not just a lead on transactions per second, but also in stability and zero-message-loss. For example, for the most basic proxy/virtualization, on average we are 2x as fast as ServiceMix and 4x as fast as Mule.

On the other side, the proprietary ESB we tested has improved significantly from last year. In fact, for the basic proxy function, the proprietary ESB has a very significant advantage.

I'd really like to see more ESB benchmarking. Firstly, I think it helps improve the whole marketplace. Secondly, since an ESB is often adding an extra hop to every interaction, ESB performance is key.

Finally, the silver lining? Well, if there is a cloud - it has to be how much we got beaten by the proprietary ESB on the proxying large messages. And if there is a silver lining - at least we can't be accused of fixing the results!