Memory problems with InputStream in Java

Question

I need to read a file into an array of bytes. The whole file needs to be read in an array. The problem is that I get an OutOfMemory error because the file size is too large. An increase in -XmX does not seem to have any effect. Here is the code snippet:

InputStream in = new FileInputStream(file); long length = file.length(); byte[] out = new byte[(int)length]; // Process the byte array 

The problem occurs while creating a byte array. Is there a less difficult workaround for this problem?

Answer 1

You need to have much more free memory than the largest file you can use for this approach. Given that a machine with 24 GB costs less than 2 thousand pounds, this is not such a stupid idea as before. In fact, the 2GB limit for byte [] in some situations is a big headache.

However, the usual way to read an InputStream is to read a block, for example, 8 KB at a time. So you need to have much more than 8k.

BTW: you can use -mx1g instead of the option you are using.

Answer 2

No, if your file is too large to fit in memory, it is too large to fit in memory.

A better solution would be to try to treat the stream as a stream, rather than loading all of this into memory. Not knowing what treatment you are trying to achieve, we cannot say if this is really possible.

For example, if you are just trying to calculate a safe hash of a file, you should do this without loading significant amounts of data at a time, but if your processing requires random access to the data, you might need to use RandomAccessFile .

Answer 3

The desktop will, rather than load the entire file into RAM. In fact, you cannot do this for large files, because you need to allocate a lot of memory in one world, which may not work.

Question ist: do you really need the whole file in memory?

change

 InputStream in = new FileInputStream(file); long length = file.length(); // At this point a warning should appear, because the code would // not work for files larger than Integer.MAX_VALUE byte[] out = new byte[(int)length]; 

Answer 4

How to use a memory mapped file: FileChannel

From http://www.java-tips.org/java-se-tips/java.nio/how-to-create-a-memory-mapped-file-3.html :

 try { File file = new File("filename"); // Create a read-only memory-mapped file FileChannel roChannel = new RandomAccessFile(file, "r").getChannel(); ByteBuffer readonlybuffer = roChannel.map(FileChannel.MapMode.READ_ONLY, 0, (int)roChannel.size()); // Create a read-write memory-mapped file FileChannel rwChannel = new RandomAccessFile(file, "rw").getChannel(); ByteBuffer writeonlybuffer= rwChannel.map(FileChannel.MapMode.READ_WRITE, 0, (int)rwChannel.size()); // Create a private (copy-on-write) memory-mapped file. // Any write to this channel results in a private // copy of the data. FileChannel pvChannel = new RandomAccessFile(file, "rw").getChannel(); ByteBuffer privatebuffer = roChannel.map(FileChannel.MapMode.READ_WRITE, 0, (int)rwChannel.size()); } catch (IOException e) { }