Effective decoding of binary and text structures (packages)

Question

Background

There is a famous tool called Wireshark . I have been using it for ages. This is great, but performance is a problem. A common use case involves several steps in preparing data to retrieve a subset of data that will be analyzed later. Without this step, it takes several minutes to filter (with large Wireshark routes near the unusable).

The real idea is to create the best solution, fast, parallel and efficient, which will be used as an aggregator / data warehouse.

Requirements

The actual requirement is to use all the power provided by modern equipment. I have to say that there is a place for various types of optimization, and I hope that I have done a good job on the upper layers, but technology is the main issue right now. In accordance with the current design, there are several options for packet decoders (dissectors):

• interactive decoders: decoding logic can be easily changed at runtime. This approach can be very useful for protocol developers - decoding speed is not critical, but flexibility and fast results are more important.
• embedded decoders: can be used as a library. This type should have good performance and be flexible enough to use all available processors and cores.
• decoders as a service: can be accessed through a clean API. This type should provide the best performance and breed efficiency.

results

My current solution is JVM-based decoders. The real idea is to reuse the code, eliminate porting, etc., but still have good performance.

• Interactive decoders: implemented on Groovy
• Embedded Decoders: Implemented in Java
• Decoders as a service: Tomcat + optimization + embedded decoders wrapped in a servlet (binary input, XML output)

Problems to be Solved

• Groovy provides a way of more power and everything, but gives expressiveness in this particular case.
• The decoding protocol in the tree structure is a dead end - too many resources are simply lost.
• Memory consumption is somewhat difficult to control. I made several optimizations, but still not happy with the profiling results.
• Tomcat with various bells and whistles still introduces a lot of overhead (mostly connection processing).

Am I using JVM correctly everywhere? Do you see any other good and elegant way to achieve your original goal: get easy-to-write scalable and efficient protocol decoders?

Protocol, format of results, etc. not fixed.

Answer 1

I found several possible improvements:

Interactive decoders

Groovy expression can be greatly improved by extending Groovy syntax using AST Transformations . Thus, it would be possible to simplify the creation of decoders that provide good performance. AST (stands for Abstract Syntax Tree) is a compile-time method.

When the Groovy compiler compiles Groovy scripts and classes, on some In this process, the source code will be presented in memory as a concrete syntax tree, and then converted to an Abstract syntax tree. The goal of AST Transformations is to allow developers to connect to the compilation process to be able to modify the AST before it becomes the bytecode that the JVM will run.

I do not want to reinvent the wheel, introducing another language to define / describe the structure of the protocol (this is enough for ASN.1 ). The idea is to simplify the development of decoders to provide some quick prototyping technology. Basically, some kind of DSL should be introduced.

Further reading

Embedded Decoders

Java may introduce some additional overhead. To solve this problem, there are several libraries:

• HPPC
• Trove
• Javolution
• Commons-primitives

Honestly, I see no other option than Java for this layer.

Decoders as a Service

At this level, Java is not required. Finally, I have a good option, but the price is pretty high. GWan looks really good.

Some additional porting will be required, but it is definitely worth it.

Answer 2

This problem, apparently, has the same characteristic of many problems with high I / O performance, which is that the number of copies of memory dominates performance. The scattering interface patterns for asynchronous I / O follow from this principle. With a scatter, memory blocks are collected. While protocol decoders accept block streams as input rather than stream streams, you should eliminate most of the overhead for moving memory to preserve the byte stream abstraction. A byte stream is a very good abstraction to save development time, but not very good for high-performance I / O.

In a related issue, I would beware of the JVM just because of the underlying String type. I cannot say that I am familiar with how String is implemented in the JVM, but I believe that there is no way to make a string from a list of blocks without making a copy of the memory. On the other hand, a native string type that could and that interacted with the JVM String compatible way could be a way to split the difference.

---

Another aspect of this problem that seems relevant is formal languages. In the spirit of not copying memory blocks, you also do not want to scan the same memory block again and again. Since you want to make changes at runtime, that means you probably don't want to use a precompiled state machine, but rather a recursive descent parser that can send the appropriate protocol interpreter at each level of descent. There are some complications associated with the fact that the outer layer does not indicate the type of inner layer. These complications are worse when you do not even get the length of the inner content, since then you rely on the inner content to be well formed to prevent escape. Nevertheless, it is worth paying attention to how many times one block will be scanned.

Answer 3

Network traffic is increasing ( some analysts ), so it will be necessary to process more and more data per second.

The only way to achieve this is to use more processor power, but the processor frequency is stable. Only the number of cores is increasing. It seems the only way is to use the available cores more efficiently and scale.