Building JavaScript streaming objects in Node.js

Question

I'm trying to wrap my head around Node.js streams, and not that I'm pretty new to JavaScript and node, the last languages ​​I really got were Perl and PHP: -D

I read the Buffer / Streams @ nodejs.org documentation, looked at James Holliday @ LXJS , read his Russian book and Torsten Lorenz event report . I'm starting to understand the basics :)

I process data that is serialized in RDF (which is neither JSON nor XML). I manage to get the data (in real code via a request) and rdfstore it into a JS object using the rdfstore module.

So far I am doing this:

 s.createReadStream('myRDFdata.ttl').pipe(serialize()).pipe(process.stdout); 

Where serialize() parses the serialization of the code at the same time. I use the through module to interact with the stream.

Now I have a few more methods (not a declaration of a real function, but I hope you get the point):

• getRecipe(parsedRDF) takes the parsed RDF (as a JavaScript object) and tells me how to use it
• createMeal(parsedRDF, recipe) → takes the parsed RDF and the recipe from above and creates a new RDF object from it
• this new object must be serialized and sent to the browser
• (In the real world, getRecipe will need to interact with the user in the browser)

I like the idea of ​​combining this together through pipes for more flexibility when I improve the code later. But I don't want to serialize it to serialize RDF every time, but just send around a JS object. From what I read in the documentation, I could use the stringify module to get a string from each step to connect it to the next step. But:

• Does it really make sense? In terms of am I adding extra overhead or is it insignificant?
• I don’t see how I could give parsedRDF to both methods with a dependency that getRecipe would need to be called, and the output will also be entered for createMeal . Are there any modules that help me with this?
• Perhaps I should ask the user to select the final recipe selection, so I may need to send the material to the browser to get the final answer. Can I do something similar on sockets while the pipe is “waiting”?

Hope this shows what I'm trying to do if I don't try to give more details / rephrase.

Update: after hibernation, I realized a few more things:

• It probably doesn't make sense to serialize the RDF format to something non-standard if there are official serialization formats. So instead of using stringify I just pass the official RDF serialization between steps
• This means that I parse / serialize the objects at each step, and this certainly adds overhead. Question: do I care? I could extend the RDF module that I use for stream analysis and serialization into one
• I can solve the problem with the dependency between getRecipe and createMeal by simply adding some information from getRecipe to parseRDF , this can be done very easily with RDF without breaking the original data model. But I would still be interested to know if I can handle dependencies like this with pipes

Answer 1

yes, it's ok to make a js object stream, you just have to remember to blow it through something that will serialize the stream again after writing it to IO.

I would suggest writing a module called rdfStream that parses and serializes rdf, you will use it as follows

 var rdf = require('rdf-stream') fs.createReadStream(file) //get a text stream .pipe(rdf.parse()) //turn it into objects .pipe(transform) //optional, do something with the objects .pipe(rdf.stringify()) //turn back into text .pipe(process.stdout) //write to IO. 

and it can also be used by other people working with rdf in node, awesome!