I hate to tell you this, but there is no easy answer to your question.
Serialization format performance depends on many factors. First of all, performance is a property of the implementation more than the format itself. What you really want to know is how well the specific JRuby implementations of each format (or maybe Java implementations, if you just wrap them) are executed. The answer can be very different from the answer in other languages, for example C ++.
In addition, performance depends on how you use the library. Many library APIs offer a trade-off between the “easy, slow” way and the “fast, hard” way. During optimization, you will want to carefully study the documentation and see a sample code from the authors of the libraries to learn how to squeeze the maximum performance.
Finally - and most importantly - performance is very different depending on the data you are working with. Different formats and implementations are optimized for different types of data. For example, string data is going to use very different code codes from data with a lot of data. For each format - even JSON and XML * - you can always find one use case where they work better than everyone else. Be careful with the criteria emanating from the authors of the libraries, as they will tend to emphasize their use cases.
Unfortunately, if you really want to know which format will work best for you, the only way to find out is to write two versions of your code using each library and comparing them. No external test can give you a real answer.
(I am the author of Protobuf v2 and Cap'n Proto , so I spent a lot of time on serialization tests and thinking about performance.)
* Just kidding about XML.