How to pass a floating point array (without serialization / deserialization) from Scala (JeroMQ) to C (ZMQ)?

Question

I am currently using the JSON library to serialize data at the sender (JeroMQ) and deserialize at the receiver (C, ZMQ). But when parsing the JSON library, it starts to consume a lot of memory, and the OS kills the process. So, I want to send the float array as is, that is, without using JSON.

The existing sender code is below ( syn0 and syn1 are Double arrays). If syn0 and syn1 are about 100 MB each, the process is destroyed when analyzing the resulting arrays, i.e. The last line of the snippet is below:

 import org.zeromq.ZMQ import com.codahale.jerkson socket.connect("tcp://localhost:5556") socket.send(json.JSONObject(Map("syn0"->json.JSONArray(List.fromArray(syn0Global)))).toString()) println("SYN0 Request sent") val reply_syn0 = socket.recv(0) println("Response received after syn0: " + new String(reply_syn0)) logInfo("Sending Syn1 request … , size : " + syn1Global.length ) socket.send(json.JSONObject(Map("syn1"->json.JSONArray(List.fromArray(syn1Global)))).toString()) println("SYN1 Request sent") val reply_syn1 = socket.recv(0) socket.send(json.JSONObject(Map("foldComplete"->"Done")).toString()) println("foldComplete sent") // Get the reply. val reply_foldComplete = socket.recv(0) val processedSynValuesJson = new String(reply_foldComplete) val processedSynValues_jerkson = jerkson.Json.parse[Map[String,List[Double]]](processedSynValuesJson) 

Is it possible to pass these arrays without using JSON?

Here I am moving a float between two C programs:

 //client.c int main (void) { printf ("Connecting to hello world server…\n"); void *context = zmq_ctx_new (); void *requester = zmq_socket (context, ZMQ_REQ); zmq_connect (requester, "tcp://localhost:5555"); int request_nbr; float send_buffer[10]; float recv_buffer[10]; for(int i = 0; i < 10; i++) send_buffer[i] = i; for (request_nbr = 0; request_nbr != 10; request_nbr++) { //char buffer [10]; printf ("Sending Hello %d…\n", request_nbr); zmq_send (requester, send_buffer, 10*sizeof(float), 0); zmq_recv (requester, recv_buffer, 10*sizeof(float), 0); printf ("Received World %.3f\n", recv_buffer[5]); } zmq_close (requester); zmq_ctx_destroy (context); return 0; } //server.c int main (void) { // Socket to talk to clients void *context = zmq_ctx_new (); void *responder = zmq_socket (context, ZMQ_REP); int rc = zmq_bind (responder, "tcp://*:5555"); assert (rc == 0); float recv_buffer[10]; float send_buffer[10]; while (1) { //char buffer [10]; zmq_recv (responder, recv_buffer, 10*sizeof(float), 0); printf ("Received Hello\n"); for(int i = 0; i < 10; i++) send_buffer[i] = recv_buffer[i]+5; zmq_send (responder, send_buffer, 10*sizeof(float), 0); } return 0; } 

Finally, my unsuccessful attempt to do something similar with Scala (below is the client code):

 def main(args: Array[String]) { val context = ZMQ.context(1) val socket = context.socket(ZMQ.REQ) println("Connecting to hello world server…") socket.connect ("tcp://localhost:5555") val msg : Array[Float] = Array(1,2,3,4,5,6,7,8,9,10) val bbuf = java.nio.ByteBuffer.allocate(4*msg.length) bbuf.asFloatBuffer.put(java.nio.FloatBuffer.wrap(msg)) for (request_nbr <- 1 to 10) { socket.sendByteBuffer(bbuf,0) } } 

Answer 1

SER / DES? The size?
No, the main limitation of transport philosophy matters.

You started with a size of 0.1 GB for the transport load and reported a JSON -library distribution to force your O / S to kill the process.

Then, in another entry, you requested 0.762 GB for the transport load.

But in the ZeroMQ transport orchestration, ZeroMQ is a slightly more important issue than choosing the policy of an external SER/DES data serializer.

No one can forbid you from trying to send as many BLOB as possible, while the JSON -decorated line has already shown you the dark side of such approaches, there are other reasons not to continue this path forward.

ZeroMQ cannot be a large and powerful toolbox. However, it takes some time to get the understanding necessary for truly smart and high-performance code deployment, which makes the most out of this powerful workhorse.

One of the side effects of the multifunctional internal ecosystem “under the hood” is the not-so-well-known policy hidden in the concept of message delivery.

You can send any message of a reasonable size, while delivery is not guaranteed. He is completely , or nothing , as stated above, nothing is guaranteed.

Uch ?!

Yes , not guaranteed.

Based on this basic philosophy of Zero-Guarrantee, due care must be taken to decide on steps and measures, especially if you plan to move Gigabyte BEASTs back and forth.

In this sense, it can be quantitatively supported by the real SUT test, that messages of a small size can be transported (if you still need to move GB (see the comment above in the OP section) and have no other choice) the entire amount of data segmented by more small pieces with errors associated with the error, resulting in a much faster and much more secure end solution than trying to use a dead end and instruct the code to dump some GB of data on any of the available resources out there (the principle of the Copy-the Zero ZeroMQ n may not be able to save you in this effort).

For more information about another hidden trap associated with the incomplete implementation of Zero-Copy, read Martin SUSTRIKA, co-father of ZeroMQ , notes on null copying "up-core-border-only" (so at least double the allocation of memory space, expected ...).

---

Decision:

Redesign your architecture to distribute small messages if you don’t keep the original datastructure mirrored in remote processes, rather than trying to keep one-time giga transfers alive.

---

Best next step?

While this does not solve the problem with several SLOC -s, the best thing is if you seriously invest your intellectual abilities in distributed processing - this is to read Pieter HINTJEN lovely book "Code Connected, Volume 1"

Yes, it takes some time to create your own understanding, but in many aspects it will lead you to another level of professional code design. Worth the time. Worth the effort.

Answer 2

You will need to serialize the data in any form or mode - in the end, you take the structure in memory on the one hand and instruct the other side on how to rebuild this structure (bonus points for using two separate languages, the structure in memory is probably does not matter). I would suggest you use the new JSON library, since it seems that the problem is that there are more efficient protocols that you could use. Protocol buffers enjoy good support in many languages, this may be the place I would start.