Apache NiFi is a reliable and secure environment for routing, converting, and delivering data across multiple systems. NiFi can work in parallel with other applications, but it works best when the entire system (or several systems in a cluster) is dedicated to it. He often uses SAN or RAID storage at the TB level for the huge amount of content that he swallows and the origin that he generates. The user interface allows multiple users to quickly change threads simultaneously on the same computer or in a cluster. The latest release of NiFi ( 1.1.0 RC1 ) includes over 170 processors for user integration with various systems and operations, and 762 MB of compression for download. In other words, NiFi is a server application.
Apache MiNiFi was developed from the recognized need to bring NiFi capabilities to the edge as “agents” - access to data from IoT and devices at the desktop level, as well as using basic NiFi capabilities at an early stage. Now, data can be collected from different protocols, have a data origin created immediately for more holistic control and transparency, have lightweight transformations applied at the source, are encrypted, prioritized and redundantly redirected to more powerful transformations performed in the cloud or data center.
Now all these behaviors can be performed using custom scripts, but then the problem of management and control (C2) arises. With hundreds, thousands, or even millions of these devices, how can anyone be monitored and complemented, and what happens when a thread needs to be changed? This may mean returning to a new endpoint, updating the frequency with which it is collected or transmitted, or to process new metrics or metadata from the device. This manual process does not scale. Due to the integration of MiNiFi with NiFi, the stream can be developed using the user interface in NiFi and transparently transferred to the MiNiFi stream and replaced by agent classes around the world.
With manual modification to remove unnecessary processors and NiFi functions, you can trim to fit on a Raspberry Pi. But this still requires the JVM, and there are many devices that do not support it. MiNiFi is offered in Java and C ++, and the size of the workspace is on a completely different scale - 39 MB for the Java agent (tar) and 310K for the C ++ agent (tar).
An excellent example of the strength and usefulness of MiNiFi is the recent TU-Automotive Detroit demo, where MiNiFi was loaded onto a Qualcomm user modem located in a "connected car." As the car drives, huge amounts of data are generated by components throughout the car and routed through CANBUS for processing. Some data is important for transferring back to the remote processing center in real time - this data is transmitted via the LTE connection. LTE is widely available, but bandwidth is expensive. Meanwhile, the data, which was much more, but less time (system diagnostics, etc.), could be packed and compressed, and then sent bursts via WiFi when the car was in the range of a known access point. MiNiFi coordinated all flow and routing solutions with feedback from geo-enrichment and control. Here is a short video by Joe Niemiec explaining the process and showing the flow .
You can extrapolate this demo to many other use cases. It is useful to think that MiNiFi is a “good guest” - a light agent who works on equipment that is probably intended for another main purpose. Regardless of whether it is an IoT, cash register / sales system, car modem, physical sensors, etc., It is not related to MiNiFi - its task is to process and expand this data without taking unnecessary resources from the main function. Contrast this with NiFi, which again can work simultaneously with other applications, but ideally it has dedicated resources that it can maximize for its own performance.