Geek Answers Handbook

Apache NiFi: Output to multiple streams simultaneously?

Is there a way to write to different streams simultaneously in a user processor in NiFi? For example, I have third-party libraries that do significant processing using APIs that work something like this:

public void process(InputStream in, OutputStream foo, OutputStream baa, List<String> args)
{
    ...
    foo.write(things);
    baa.write(stuff);
    ...
}

But the only examples I can find just use one output stream:

FlowFile transform = session.write(original, new OutputStreamCallback() {
        @Override
        public void process(OutputStream out) throws IOException {
            out.write("stuff");
        }
    });

Processing is performed in batches (due to the large scale), so it is impractical to perform all the processing, and then write out individual threads.

The only way I can come up with is the input process several times :(

To clarify, I want to write several FlowFiles using the method session.write(flowfile, callback), so different threads can be sent / managed separately

+4
2

API NiFi , - :

        FlowFile flowFile1 = session.create();
        final AtomicReference<FlowFile> holder = new AtomicReference<>(session.create());

        flowFile1 = session.write(flowFile1, new OutputStreamCallback() {
            @Override
            public void process(OutputStream out) throws IOException {

                FlowFile flowFile2 = session.write(holder.get(), new OutputStreamCallback() {
                    @Override
                    public void process(OutputStream out) throws IOException {

                    }
                });
                holder.set(flowFile2);

            }
        });
+4

, , , . "" "", , , DoThings DoStuff. , . /etc. NiFi - , , .

+3

More articles:


All Articles