Has anyone used data stream programming in a real project with a main language?

Question

I am considering using some of the data flow programming techniques in the clojure program, but I find it difficult to find a lot of information from projects using Java, C # or other main languages ​​that used such methods in the real world. I would appreciate hearing if anyone has any experience with whom they could share this.

Answer 1

Here we are! We did ... (quote from one of my old posts):

We developed and implemented a DF server for our automation project (dispatcher, component interface, set of components, DF language, DF compiler, user interface). It is written in bare C ++, and works on several Unix-like systems (Linux x86, MIPS, avr32, etc. Mac OS X). It lacks several functions, for example, complex flow control, complex flow control (there is only a not too advanced component for it), so this is just a prototype, even it works. We are currently working on a fully functional server. We learned a lot during the implementation and use of the prototype.

In addition, we will create a visual editor of the day.

There are data flow systems that do not even mention the approach to data flow:

• SynthEdit: http://www.synthedit.com/ - These are related audio frames and a set of components for creating VST plugins
• TinyOS: http://www.tinyos.net/ - it’s an embedded operating system / framework
• Digital synthetisers / samplers of system data streams programmed - presumably - in C or some parts in the Assembly, check out my answer to another post about some examples
• Quartz Composer , a graphical magic tool for Mac,
• Blender has a data flow subsystem for composing images.

Writing a data flow system is not rocket science. Here's my older article on the basics of data flow structure .

The term data stream is broad. There are synchronous real-time data flow systems, such as synthetic and samplers, there are asynchronous ones, for example, our home aut. (the system is in standby mode if the user does not press a button or the timer does not end), and there are even different architectures such as spreadsheets or do.

Want to know more about data flow programming? Read the J. Paul Morrison website and book .

Answer 2

Pervasive DataRush is the foundation for parallel data flow programming for any JVM language, including Clojure.

Pervasive DataRush uses a data stream architecture. The architecture implements a program that runs in the form of a graph of computation nodes interconnected by queues of data streams. Nodes use queues to exchange data. As data is streamed, only the data required by any active operation must be in memory at any given time, which allows the analysis of very large data sets. In addition to the ability to scale for tasks larger than the available memory, data flow diagrams use several forms of parallelism.

Customers use DataRush for large data analytics and data preparation (ETL).

Answer 3

We did another one: a joint spreadsheet with the MySQL / PHP backend and AJAX interface. The software is in beta, documentation is under development.