C ++ Protocol Buffer, sending an integer array

Question

I have a built-in C ++ project where I read an int32 series from a hardware device and then pack them into an int array as part of a large data structure and then send it to the remote system via TCP / IP. So, I used a simple data structure with specific data sets, and now I want to convert this to using protocol buffers. So, I was thinking about using "Int32 duplicate data" as an element of my proto buff. But I want to avoid using a loop, for example:

int hardware_data[1000]; // An array that holds the data read from the hardware for(int i=0; i< sizeof(hardware_data); i++ ) { proto.add_data( hardware_data[i] ); } 

I would rather use an efficient method, for example, so that proto buff simply points to an existing hardware_data [] array (null copy method) or use memcpy from hardware_data in proto.data.

I understand how to configure memcpy (), but how then does proto buff know how many elements are in the proto.data array? Can I use proto.data_size () to get the number of elements? Is there an efficient way to move data from my equipment read in proto buff to send? Is there a better way to do this?

Kerrik, I did not know about the zero copy API. Here is my definition of proto:

 message hardware_data { optional Lob lob = 1; optional int32 taskSeqNum = 2; optional int32 secondsOfDay = 3; optional float IQOutRateKhz = 4; optional float IQBwKhz = 5; optional int32 tStart = 6; optional int32 tOffset = 7; optional float collectionTime = 8; optional int32 numSamples = 9; optional int32 chunk = 10; optional int32 dimSize = 11; repeated int32 data = 12 [packed=true]; } 

I'm not sure how a null copy will play this definition of proto buff.

Answer 1

On the wire, a packaged repeated int32 is encoded as a series of varieties. Varint is a variable-width encoding in which smaller values ​​take up less space. Of course, this is not how the data is represented in your array, so nesting it in a null copy of the message is actually impossible.

In fact, you are currently making two copies, and you can eliminate one of them. Instead of directly allocating int hardware_data[1000] consider inserting data directly into google::protobuf::RepeatedField<int> . Then you can make smart use of Swap() to move this data to the message without a copy:

 RepeatedField<int> hardware_data; hardware_data.Reserve(expected_size); get_data_somehow(&hardware_data); // later proto.mutable_data()->Swap(&hardware_data); 

After you have serialized the message, you might want to further change () the field back so that you can reuse the stored memory. ( RepeatedField::Clear() will not free up base memory, just mark it for reuse.)

For all that serializing a message will still require copying the data as part of its encoding. Even if you change the encoding to a fixed32 packed multiplier (which is actually encoded as 32-bit integers on the wire), there is no way to convince the library to use your memory directly.