Geek Answers Handbook

Line streams in Node.js

I am developing a multiprocessor application using Node.js. In this application, the parent process spawns a child process and communicates with it using the JSON messaging protocol over the pipe. I found that large JSON messages can be "truncated", so that one "piece" emitted by the data listener on the handset does not contain a complete JSON message. In addition, small JSON messages can be grouped into the same fragment. Each JSON message will be limited to a newline character, and so I wonder if there is already a utility that will buffer the feed stream of the channel so that it emits one line at a time (and therefore, for my application, one JSON document in time) . This seems to be a fairly common use case, so I wonderif it is already done.

I would be grateful for any guidance that anyone can offer. Thank.

+5
source share
4 answers

Maybe Pedro carrier can help you?

The carrier helps you implement a new line of completed protocols through node.js.

The client can send you pieces of the line and the carrier will notify you only on each completed line.

+4
source

My solution to this problem is to send JSON messages, each of which ends with a special Unicode character. A character you'll never get in a JSON string. Call it TERM.

, "JSON.stringify(message) + TERM;" . TERM JSON.parse(), . , , , . :

        s.on("data", function (data) {
        var info = data.toString().split(TERM);
        info[0] = fragment + info[0];
        fragment = '';

        for ( var index = 0; index < info.length; index++) {
            if (info[index]) {
                try {
                    var message = JSON.parse(info[index]);
                    self.emit('message', message);
                } catch (error) {
                    fragment = info[index];
                    continue;
                }
            }
        }
    });

"" - , .

? Unicode '\ uFFFD'. , , "\ r\n", "\n" "\ r\n"

, , ..

+2

- json- (4 ?) , .

You can try node-binary to avoid writing the parser manually. Check out the sample documentation scan(key, buffer)- it does the exact reading of the string.

+1
source

While newlines (or any other separator you use) will restrict JSON messages and not be embedded in them, you can use the following pattern:

const buf = ''
s.on('data', data => {
  buf += data.toString()
  const idx = buf.indexOf('\n')
  if (idx < 0) { return } // No '\n', no full message
  let lines = buf.split('\n')
  buf = lines.pop() // if ends in '\n' then buf will be empty
  for (let line of lines) {
    // Handle the line
  }
})
0
source

More articles:


All Articles