Passing a structure over TCP (SOCK_STREAM) socket in C

Question

I have a small client server application in which I want to send the entire structure via a TCP socket in C not C ++. Suppose the struct is as follows:

struct something{ int a; char b[64]; float c; } 

I found many posts saying that I need to use the pragma package or serialize the data before sending and receiving.

My question is: is it enough to use the JUST pragma pack or just a series? Or do I need to use both?

In addition, since serialization is a process with an intensive processor, this significantly reduces the efficiency of your work, so what is the best way to serialize the structure WITHOUT using an external library (I would like to try a sample code / algorithm)?

Answer 1

For network portability, you need the following:

• Pack the structure. For gcc and compatible compilers, do this with __attribute__((packed)) .

• Do not use any elements except unsigned integers of a fixed size, other packed structures that satisfy these requirements, or arrays of any of them. Signed integers are also OK, unless your machine uses a two-pad representation.

• Decide whether your protocol will use encodings with more or more integers. Make conversions when reading and writing these integers.

• In addition, do not accept pointers to elements of the packed structure , except those that have a size of 1 or other nested packed structures. See this answer .

The following is a simple example of encoding and decoding. It assumes that the byte conversion functions hton8() , ntoh8() , hton32() and ntoh32() (the first two are non-op, but there for consistency).

 #include <stdint.h> #include <inttypes.h> #include <stdlib.h> #include <stdio.h> // get byte order conversion functions #include "byteorder.h" struct packet { uint8_t x; uint32_t y; } __attribute__((packed)); static void decode_packet (uint8_t *recv_data, size_t recv_len) { // check size if (recv_len < sizeof(struct packet)) { fprintf(stderr, "received too little!"); return; } // make pointer struct packet *recv_packet = (struct packet *)recv_data; // fix byte order uint8_t x = ntoh8(recv_packet->x); uint32_t y = ntoh32(recv_packet->y); printf("Decoded: x=%"PRIu8" y=%"PRIu32"\n", x, y); } int main (int argc, char *argv[]) { // build packet struct packet p; px = hton8(17); py = hton32(2924); // send packet over link.... // on the other end, get some data (recv_data, recv_len) to decode: uint8_t *recv_data = (uint8_t *)&p; size_t recv_len = sizeof(p); // now decode decode_packet(recv_data, recv_len); return 0; } 

Regarding byte order conversion functions, your htons() / ntohs() and htonl() / ntohl() can be used for 16- and 32-bit integers, respectively, for converting to / from large byte ordering. However, I do not know any standard function for 64-bit integers or for converting to / from little endian. You can use my byte conversion functions ; if you do, you must specify the byte order of your computer by specifying BADVPN_LITTLE_ENDIAN or BADVPN_BIG_ENDIAN .

As for signed integers, conversion functions can be implemented safely in the same way as those that I wrote and linked (by directly changing bytes); just replace unsigned with the signed one.

UPDATE : if you want to use an efficient binary protocol, but don’t like to mess with bytes, you can try something like Protocol Buffers ( C-implementation ). This allows you to describe the format of your messages in separate files and generate the source code that you use to encode and decode messages of the format you specify. I also implemented something similar to myself, but greatly simplified; see my BProto generator and a few examples (see .bproto and addr.h for a usage example).

Answer 2

Before sending any data over a TCP connection, define the protocol specification. This should not be a multi-page document filled with technical jargon. But he must indicate who sends what and when he should indicate all messages at the byte level. He must indicate how the ends of the messages are set, whether there are timeouts and who imposes them, etc.

Without specification, it is easy to ask questions that are simply impossible to answer. If something goes wrong, which end is to blame? With the specification, an error that does not meet the specification is to blame. (And if both ends are compliant and still not working, the spec is to blame.)

Once you have a specification, it is much easier to answer questions about how one end or the other should be designed.

I also strongly recommend that you do not develop a network protocol that is specific to your equipment. At least not without a proven performance problem.

Answer 3

It depends on whether you can be sure that your systems at both ends of the connection are homogeneous or not. If you are sure at all times (which most of us cannot be), you can take a few shortcuts - but you should know that they are shortcuts.

 struct something some; ... if ((nbytes = write(sockfd, &some, sizeof(some)) != sizeof(some)) ...short write or erroneous write... 

and similar read() .

However, if there is a possibility that the systems may be different, you need to establish how the data will be transferred formally. You can linearize (serialize) the data - perhaps with love with something like ASN.1, or perhaps more simply with a format that can be easily re-read. For this, the text is often beneficial - it is easier to debug when you can see what is going wrong. Otherwise, you need to determine the byte order in which the int is transmitted, and make sure that the transfer matches that order, and the line will probably get the number of bytes followed by the corresponding amount of data (think whether the terminal should be null or not), and then some view of the float. This is weirder. It is not easy to write serialization and deserialization functions to handle formatting. The hard part is the development (decision) of the protocol.

Answer 4

You can use union with the structure you want to send and an array:

 union SendSomething { char arr[sizeof(struct something)]; struct something smth; }; 

Thus, you can send and receive only arr. Of course, you have to take care of the problems with endianess, and sizeof(struct something) can vary on different machines (but you can easily overcome this with the #pragma pack ).

Answer 5

Why would you do this when there are good and fast serialization libraries like the Message Pack that do all the hard work for you, and as a bonus they provide you with cross-language compatibility of your socket protocol?

To do this, use Message Pack or another serialization library.

Answer 6

Typically, serialization brings several advantages, for example, sending bits of a structure over a wire (for example, fwrite ).

• This happens individually for each non-aggregate atomic information (for example, int).
• It accurately determines the serial data format sent by cable
• Thus, he deals with heterogeneous architecture: sending and receiving machines can have different lengths and final word lengths.
• It may be less fragile if the type changes a little. Therefore, if an old version of your code is installed on one machine, it can talk with a machine with a newer version, for example. one of which has char b[80]; instead of char b[64];
• It can deal with more complex data structures - variable vector sizes or even hash tables - in a logical way (for a hash table, pass an association, ..)

Very often serialization procedures are generated. 20 years ago, RPCXDR already existed for this purpose, and XDR serialization primitives are still in many libc.

Answer 7

The Pragma package is used for binary compatibility of your structure at the other end. Since the server or client to which you send the structure can be written in another language or created using another c compiler or with other parameters of the c compiler.

Serialization, as I understand it, makes a stream of bytes from your structure. When you write a structure on a socket, you do serialization.

Answer 8

If you need portability, then you must serialize each element separately due to the completion of line and structure input.

Here is an example of using Binn :

  binn *obj; // create a new object obj = binn_object(); // add values to it binn_object_set_int32(obj, "id", 123); binn_object_set_str(obj, "name", "Samsung Galaxy Charger"); binn_object_set_double(obj, "price", 12.50); binn_object_set_blob(obj, "picture", picptr, piclen); // send over the network send(sock, binn_ptr(obj), binn_size(obj)); // release the buffer binn_free(obj); 

These are just 2 files (binn.c and binn.h), so you can compile it using a project instead of using it as a shared library.

Perhaps you should also use the message framework (also known as length prefix framing) in socket streams.

Answer 9

The Google Buffer protocol offers an excellent solution to this problem. See Here Google Protobol Buffer - C Implementation

Create a .proto file based on your payload structure and save it as payload.proto

  syntax="proto3" message Payload { int32 age = 1 string name = 2 } . 

Compile the .proto file using

  protoc --c_out=. payload.proto 

This will create the payload.pb-ch header file and its corresponding payload.pb-cc in your directory.

Create your server.c file and include the protobuf-c header files

 #include<stdio.h> #include"payload.pb.ch" int main() { Payload pload = PLOAD__INIT; pload.name = "Adam"; pload.age = 1300000; int len = payload__get_packed_size(&pload); uint8_t buffer[len]; payload__pack(&pload, buffer); // Now send this buffer to the client via socket. } 

On your receiving side client.c

 .... int main() { uint8_t buffer[MAX_SIZE]; // load this buffer with the socket data. size_t buffer_len; // Length of the buffer obtain via read() Payload *pload = payload_unpack(NULL, buffer_len, buffer); printf("Age : %d Name : %s", pload->age, pload->name); } 

Make sure you compile your programs with the -lprotobuf-c flag

 gcc server.c payload.pb-cc -lprotobuf-c -o server.out gcc client.c payload.pb-cc -lprotobuf-c -o client.out