Java code to write a long array to a file

Question

I got the code here to write an Int array to a file. However, I am trying to convert it so that it can write a long array to a file. But it gives an error (the code below). Can someone help me why it gives an error and what the correct code should be. Thanks.

import java.io.*; import java.util.ArrayList; import java.io.File; import java.io.FileInputStream; import java.io.FileOutputStream; import java.io.IOException; import java.nio.ByteBuffer; import java.nio.channels.FileChannel; import java.util.Random; import java.util.zip.DataFormatException; import java.util.zip.Deflater; import java.util.zip.Inflater; public class Test { private static final int bucketSize = 1<<17;//in real world should not be const, but we bored horribly static final int zipLevel = 2;//feel free to experiement, higher compression (5+)is likely to be total waste static void writes(long[] a, File file, boolean sync) throws IOException{ byte[] bucket = new byte[Math.min(bucketSize, Math.max(1<<13, Integer.highestOneBit(a.length >>3)))];//128KB bucket byte[] zipOut = new byte[bucket.length]; final FileOutputStream fout = new FileOutputStream(file); FileChannel channel = fout.getChannel(); try{ ByteBuffer buf = ByteBuffer.wrap(bucket); //unfortunately java.util.zip doesn't support Direct Buffer - that would be the perfect fit ByteBuffer out = ByteBuffer.wrap(zipOut); out.putLong(a.length);//write length aka header if (a.length==0){ doWrite(channel, out, 0); return; } Deflater deflater = new Deflater(zipLevel, false); try{ for (int i=0;i<a.length;){ i = puts(a, buf, i); buf.flip(); deflater.setInput(bucket, buf.position(), buf.limit()); if (i==a.length) deflater.finish(); //hacking and using bucket here is tempting since it copied twice but well for (int n; (n= deflater.deflate(zipOut, out.position(), out.remaining()))>0;){ doWrite(channel, out, n); } buf.clear(); } }finally{ deflater.end(); } }finally{ if (sync) fout.getFD().sync(); channel.close(); } } static long[] reads(File file) throws IOException, DataFormatException{ FileChannel channel = new FileInputStream(file).getChannel(); try{ byte[] in = new byte[(int)Math.min(bucketSize, channel.size())]; ByteBuffer buf = ByteBuffer.wrap(in); channel.read(buf); buf.flip(); long[] a = new long[(int)buf.getLong()]; if (a.length==0) return a; int i=0; byte[] inflated = new byte[Math.min(1<<17, a.length*4)]; ByteBuffer intBuffer = ByteBuffer.wrap(inflated); Inflater inflater = new Inflater(false); try{ do{ if (!buf.hasRemaining()){ buf.clear(); channel.read(buf); buf.flip(); } inflater.setInput(in, buf.position(), buf.remaining()); buf.position(buf.position()+buf.remaining());//simulate all read for (;;){ int n = inflater.inflate(inflated,intBuffer.position(), intBuffer.remaining()); if (n==0) break; intBuffer.position(intBuffer.position()+n).flip(); for (;intBuffer.remaining()>3 && i<a.length;i++){//need at least 4 bytes to form an int a[i] = intBuffer.getInt(); } intBuffer.compact(); } }while (channel.position()<channel.size() && i<a.length); }finally{ inflater.end(); } // System.out.printf("read ints: %d - channel.position:%d %n", i, channel.position()); return a; }finally{ channel.close(); } } private static void doWrite(FileChannel channel, ByteBuffer out, int n) throws IOException { out.position(out.position()+n).flip(); while (out.hasRemaining()) channel.write(out); out.clear(); } private static int puts(long[] a, ByteBuffer buf, int i) { for (;buf.hasRemaining() && i<a.length;){ buf.putLong(a[i++]); } return i; } private static long[] generateRandom(int len){ Random r = new Random(17); long[] n = new long [len]; for (int i=0;i<len;i++){ n[i]= r.nextBoolean()?0: r.nextInt(1<<23);//limit bounds to have any sensible compression } return n; } public static void main(String[] args) throws Throwable{ File file = new File("xxx.xxx"); long[] n = generateRandom(3000000); //{0,2,4,1,2,3}; long start = System.nanoTime(); writes(n, file, false); long elapsed = System.nanoTime() - start;//elapsed will be fairer if the sync is true System.out.printf("File length: %d, for %d ints, ratio %.2f in %.2fms %n", file.length(), n.length, ((double)file.length())/4/n.length, java.math.BigDecimal.valueOf(elapsed, 6) ); long[] m = reads(file); //compare, Arrays.equals doesn't return position, so it sucks/kinda for (int i=0; i<n.length; i++){ if (m[i]!=n[i]){ System.err.printf("Failed at %d%n",i); break; } } System.out.printf("All done!"); }; } 

Answer 1

So, I took a few minutes to actually run the code, and it took a little change from your published code, but here it is.

The only thing I did, it was - optionally - was changing intBuffer to longBuffer , just for clarity. This is part of the first difference.

 75 - byte[] inflated = new byte[Math.min(1<<17, a.length*4)]; 76 - ByteBuffer intBuffer = ByteBuffer.wrap(inflated); 76 + byte[] inflated = new byte[Math.min(1<<17, a.length*8)]; 77 + ByteBuffer longBuffer = ByteBuffer.wrap(inflated); 

In the above snippet, I change the length of the inflated buffer to a.length * 8 to reflect it as a long array, not an int array.

 89 - int n = inflater.inflate(inflated,intBuffer.position(), intBuffer.remaining()); 90 + int n = inflater.inflate(inflated,longBuffer.position(), longBuffer.remaining()); 

This is just a variable name change.

 92 - intBuffer.position(intBuffer.position()+n).flip(); 93 - for (;intBuffer.remaining()>3 && i<a.length;i++){//need at least 4 bytes to form an int 94 - a[i] = intBuffer.getInt(); 93 + longBuffer.position(longBuffer.position()+n).flip(); 94 + for (;longBuffer.remaining()>7 && i<a.length;i++){//need at least 4 bytes to form an int 95 + a[i] = longBuffer.getLong(); 

This is a very important change. The name was changed at first, but this is not an important part. Secondly, remaining() is 7 instead of 3, as mentioned above. Finally, a [i] now gets the length instead of int. . This is the biggest problem, of course.

 96 - intBuffer.compact(); 97 + longBuffer.compact(); 

Just a rename here.

 142 - System.out.printf("File length: %d, for %d ints, ratio %.2f in %.2fms %n", file.length(), n.length, ((double)file.length())/4/n.length, java.math.BigDecimal.valueOf(elapsed, 6) ); 143 + System.out.printf("File length: %d, for %d ints, ratio %.2f in %.2fms %n", file.length(), n.length, ((double)file.length())/8/n.length, java.math.BigDecimal.valueOf(elapsed, 6) ); 

This is only the output of the file, to get an idea of ​​compression, now it calculates the number of results from the .length / 8 file instead of more than 4.

And these are the only necessary changes that I had to make to make it work. Basically just moving from int to long in all places.

The full code is here in pastebina, in case you mess up the diff designation or something else: http://pastebin.com/emY14Ji4

Note: the line numbers in my copy (+) are higher than your copy (-) due to a debugging instruction that I did not delete ... whoops ...

Answer 2

Do not use a byte buffer; instead, use an ObjectOutputStream to store a valid file and an ObjectInputStream to read from it. I believe the problem is that you do not keep a “long” definition throughout the process.

ObjectOutputStream and ObjectInputStream can read / write any data types.