Xuggle cannot open memory entry

Question

I am working on a program that integrates the Hadoop MapReduce framework with Xuggle. To do this, I implement the IURLProtocolHandlerFactory class, which reads and writes from Hadoop data objects to and from memory.

Here you can see the corresponding code: https://gist.github.com/4191668

The idea is to register each BytesWritable object in the IURLProtocolHandlerFactory class using a UUID so that when I later refer to this name when I open the file, it returns an instance of IURLProtocolHandler that is attached to this BytesWritable object and I can read and write from and in memory.

The problem is that I get an exception like this:

 java.lang.RuntimeException: could not open: byteswritable:d68ce8fa-c56d-4ff5-bade-a4cfb3f666fe at com.xuggle.mediatool.MediaReader.open(MediaReader.java:637) 

(see also link)

When debugging, I see that the objects are correctly found in the factory, what else, they are even read from the protocol handler. If I delete listeners from / to the output file, this happens, so the problem is with the input. Digging deeper into the Xuggle code, I get to the JNI code (which tries to open the file), and I can't get any further. This obviously returns an error code.

 XugglerJNI.IContainer_open__SWIG_0 

I am very grateful for a hint where to go next, how can I continue debugging. My implementation may have a flaw, but I don't see it.

Answer 1

I think the problem you are facing is that many types of inputs / outputs are converted into their own file descriptor into IContainer JNI code, but what you pass cannot be converted. It may not be possible to create your own IURLProtocolHandler this way, because after traveling through XuggleIO.map() he can call IContainer again and then JNI IContainer code, which will probably try to get its own descriptor file and call avio_open() .

However, there may be a few things that you can open in IContainer that are not files / do not have file descriptors, and that will be handled correctly. What you can open can be seen in IContainer , namely java.io.DataOutput and java.io.DataOutputStream (and the corresponding inputs). I recommend making your implementation of DataInput / DataOutput, which wraps around BytesReadable / BytesWriteable and opens it in IContainer.

If this does not work, write your entries in the temp file and read the outputs from the temporary file :)

Answer 2

First you can copy the file to the local one, and then try opening the container:

 filePath = split.getPath(); final FileSystem fileSystem = filePath.getFileSystem(job); Path localFile = new Path(filePath.getName()); fileSystem.createNewFile(localFile); fileSystem.copyToLocalFile(filePath, localFile); int result = container.open(filePath.getName(), IContainer.Type.READ, null); 

This code works for me in the RecordReader class.

In your case, you can first copy the file to a local one, and then try to create a MediaReader