Best way to serialize a C structure for Java deserialization, etc.

Question

I am currently saving and loading some data into C / C ++ structures in files using fread () / fwrite (). This works great when working in this C application (I can recompile whenever the structure is changed to update the sizeof () arguments for fread () / fwrite ()), but how can I load this file into other programs without knowing sizeof in advance () s of structure C?

In particular, I wrote this other Java application that visualizes the data contained in this binary C struct binary, but I would like to get a general solution on how to read this binary. (Instead of manually inserting me into sizeof () s into the source of the Java application whenever the C structure changes ...)

I'm thinking of serializing for text or XML of some kind, but I'm not sure where to start (how to serialize in C, then how to deserialize in Java and possibly other languages ​​in the future), and if appropriate here, when one member of the structure is a floating-point array that can grow up to ~ 50 MB in binary format (and I have hundreds of these data files for reading and writing).

The structure of C is simple (there are no strict references to nesting or a pointer) and looks like this:

struct MyStructure { char *title; int id; int param1; int param2; float *data; } 

The part that can change the majority are param integers.

What are my options here?

Answer 1

You can use the Java DataInput / DataOutput format, which is well described in javadoc .

Answer 2

If you have control over both codebases, you should use Protocol Buffers .

Answer 3

Take a look at JSON. http://www.json.org . If you go with javascript this will help. I do not know how good Java support is.

Answer 4

If your structure does not change (significantly), and your data is in a fairly consistent format, you can simply write the values ​​to a CSV file or some other simple format.

This can be easily read in Java, and you don’t have to worry about serializing in XML. Sometimes a simple route is the easiest route.

Answer 5

See Hessian / Burlap Resin Services . Perhaps you do not need the entire service, just part of the API and an understanding of the wired protocol.

Answer 6

One of the possibilities is to create small XML files with a header, identifier, parameters, etc., and then a link (by file name) that contains floating point data. Assuming there is no special data about floating data, and that Java and C use the same floating-point format, you can read this file using readFloat () DataInputStream .

Answer 7

I like the answers of CSV and “Protocol Protocols” (although, at first glance, the protocol buffer buffer can be very similar to YAML for everyone that I know).

If you need tightly packed records for high-volume data, you might think about this:

Create a text file header that describes the current file structure: record sizes (types?) And field names / sizes. Read and analyze the header, then use low-level I / O to load each record field, er, object properties, or what we call this year.

This gives you the opportunity to slightly modify the structure of the structure and provide its self-description, while at the same time allowing you to pack a large amount in less space than XML allows.

TMTOWTDI, I think.

Answer 8

If a:

• your data is essentially a large array of floats;
• You can test the write / read procedure in all possible environments (= machine combinations / OS / C compiler) on which each end will work;
• performance is important.

then I probably just continue to write data from C the way you do it (maybe with a little correction - see below) and turn the problem into the way you read this data from Java.

To read data from Java, use ByteBuffer . Essentially, get the tiles out of your data, wrap the ByteBuffer around them, and then use the get (), getFloat (), getInt () methods, etc. The NIO package also has buffer wrappers, for example. FloatBuffer, which of the tests that I did is apparently about 20% faster for reading large numbers of the same type.

Now you need to be careful, this is the byte order. Starting with Java, you must call order (ByteOrder.LITTLE _ ENDIAN) or order (ByteOrder.BIG _ ENDIAN) in your buffer before reading data. To decide what to use, I would recommend that at the very beginning of the stream you write a known value of 16 bytes (for example, 255 = 0x00ff). Then, pull these two bytes from Java and check the order (0xff, 0x00 or 0x00, 0xff) to see if you have a small or big end.