Quickly read small integers from a file

Question

I need to read a binary file consisting of 4 byte integers (small end) in a 2D array for my Android application. My current solution is as follows:

DataInputStream inp = null; try { inp = new DataInputStream(new BufferedInputStream(new FileInputStream(procData), 32768)); } catch (FileNotFoundException e) { Log.e(TAG, "File not found"); } int[][] test_data = new int[SIZE_X][SIZE_Y]; byte[] buffer = new byte[4]; ByteBuffer byteBuffer = ByteBuffer.allocate(4); for (int i=0; i < SIZE_Y; i++) { for (int j=0; j < SIZE_X; j++) { inp.read(buffer); byteBuffer = ByteBuffer.wrap(buffer); test_data[j][SIZE_Y - i - 1] = byteBuffer.order(ByteOrder.LITTLE_ENDIAN).getInt(); } } 

This is pretty slow for a 2k * 2k array, it takes about 25 seconds. I see in DDMS that the garbage collector is working overtime, so this is probably one of the reasons for the slowdown.

There should be a more efficient way to use ByteBuffer to read this file in an array, but I don't see it at the moment. Any idea on how to speed this up?

Answer 1

Why not read into a 4-byte buffer, and then manually change the bytes? It will look like this:

 for (int i=0; i < SIZE_Y; i++) { for (int j=0; j < SIZE_X; j++) { inp.read(buffer); int nextInt = (buffer[0] & 0xFF) | (buffer[1] & 0xFF) << 8 | (buffer[2] & 0xFF) << 16 | (buffer[3] & 0xFF) << 24; test_data[j][SIZE_Y - i - 1] = nextInt; } } 

Of course, it is assumed that read reads all four bytes, but you should check the situation when this is not the case. This way you won’t create any objects while reading (so don’t bother with the garbage collector), you do not call anything, you just use bitwise operations.

Answer 2

If you are on a platform that supports memory mapped files, consider MappedByteBuffer and friends from java.nio

 FileChannel channel = new RandomAccessFile(procData, "r").getChannel(); MappedByteBuffer map = channel.map(FileChannel.MapMode.READ_ONLY, 0, 4 * SIZE_X * SIZE_Y); map.order(ByteOrder.LITTLE_ENDIAN); IntBuffer buffer = map.asIntBuffer(); int[][] test_data = new int[SIZE_X][SIZE_Y]; for (int i=0; i < SIZE_Y; i++) { for (int j=0; j < SIZE_X; j++) { test_data[j][SIZE_Y - i - 1] = buffer.get(); } } 

If you need cross-platform support, or if your platform does not have enough mappable buffers, you can still avoid using the conversions themselves with IntBuffer. Consider dropping a BufferedInputStream by distributing the larger ByteBuffer yourself and getting a little-endian view of the IntBuffer data. Then, in the reset loop, the buffer is placed at 0, use DataInputStream.readFully to read large areas immediately in ByteBuffer and pull int values ​​from IntBuffer.

Answer 3

First of all, your "inp.read (buffer)" is unsafe because the read contract does not guarantee that it will read all 4 bytes.

Aside, for fast transformation, use the algorithm from DataInputStream.readInt

I adapted the case of a byte array of 4 bytes for you:

 int little2big(byte[ ] b) { return (b[3]&0xff)<<24)+((b[2]&0xff)<<16)+((b[1]&0xff)<<8)+(b[0]&0xff); } 

Answer 4

I don’t think it is necessary to reinvent the wheel and re-execute the byte order for the byte order. This is error prone, and there is a reason there is a class like ByteBuffer .

Your code can be optimized in the sense that it wastes objects. When byte[] wraps around a ByteBuffer , the buffer adds the view, but the original array remains the same. It doesn't matter when the source array is modified / read directly or an instance of ByteBuffer .

Therefore, you only need to initialize one instance of ByteBuffer , and also install ByteOrder .

To start again, simply use rewind() to set the counter back to the beginning of the buffer.

I took your code and changed it as described. Keep in mind that it does not check for errors if there are not enough bytes left at the input. I would suggest using inp.readFully as it will EOFException if not enough bytes are found to fill the buffer.

 int[][] test_data = new int[SIZE_X][SIZE_Y]; ByteBuffer byteBuffer = ByteBuffer.wrap(new byte[4]).order(ByteOrder.LITTLE_ENDIAN); for (int i=0; i < SIZE_Y; i++) { for (int j=0; j < SIZE_X; j++) { inp.read(byteBuffer.array()); byteBuffer.rewind(); test_data[j][SIZE_Y - i - 1] = byteBuffer.getInt(); } }