Java - How to write a very large (20,000x20,000 px or more) tif image

Question

I work with extremely large tif images that I compose in a large single image. I have a library created by my colleague that generates a pyramid of images and provides a very convenient tool for visualizing a pyramid of images. This visualizer is great for picking out a peak in a large image and visually identifying points of interest, but customers are more interested in analyzing the images in these large images.

Thus, it is necessary to export a very large image into a single file. I find this to be unpleasant given that these images can range from 800 MB to several GB. And the task of loading this single image into memory is complicated, especially when image analysis is performed.

I was wondering if it would be possible in java to write this large tiff image in a block or in turn. My app currently runs out of memory on small (8 GB RAM) machines.

The current method of compiling these images is:

• Save pixel values ​​in BufferedImage with WritableRaster

   short[][]pixels = ... BufferedImage image = new BufferedImage(width, height, type); WritableRaster = image.getRaster(); for (int row = 0; row < height; row++) { for (int col = 0; col < width; col++) { raster.setSample(col, row, 0, pixel[row][col]); } } 

• Then write the buffer image to disk. For this part, I use ImageJ to write the image as tif. If there are more efficient ways to support 16-bit grayscale images, then I’m happy to see

   // BufferedImage image; from above ... ImagePlus img = new ImagePlus(); img.setImage(image); FileSaver fs = new FileSaver(img); fs.saveAsTiff(file.getAbsolutePath()); 

The problem with this method is that it has too much memory for 8 GB of RAM.

Ideally, I would like for me to have only one short[][]pixels . This is mainly because I need to calculate the average mixing function, so there will be some amount of memory. Also in the future I will add a linear mixture. short[][]pixels should only accept ~ 765 MB of RAM for 20 kg pixel data, which I think is currently unavoidable, so for large images like 100k x 100k pixels, I hope that biologists don’t want to export this one since it takes 18 GB of RAM.

Later, I will modify the code to support the export of extremely large images such as 100k x 100k. So far, I agree with the assumption that one piece of memory will retain the initial pixel values.

So, which is a good method for writing portions of a tif image to disk, so I can support writing from basic images such as 100k x 100k images.

I saw this post: Write TILF output using ImageIO in Java

But he just discusses the TIFF 6.0 specification. But I will consider ImageOutputStreams . Tif is a beast, although I can just bite a bullet and encourage biologists to export only regions of interest.

EDIT: Viable solution found:

WRITER: https://github.com/openmicroscopy/bioformats/blob/v4.4.8/components/scifio/src/loci/formats/out/TiffWriter.java

and

READER: https://github.com/openmicroscopy/bioformats/blob/v4.4.8/components/scifio/src/loci/formats/in/TiffReader.java

Group homepage: https://github.com/openmicroscopy/bioformats

Answer 1

Ok, I found a good solution for this in Java.

Thanks to @bdares for pointing out BigTiff.

But complete with FIJI there is a group of bio-formats that implemented scifio.

They provide a number of supported readers / writers, one of which is TiffReader / Writer

https://github.com/openmicroscopy/bioformats/blob/v4.4.8/components/scifio/src/loci/formats/in/TiffReader.java

and

https://github.com/openmicroscopy/bioformats/blob/v4.4.8/components/scifio/src/loci/formats/out/TiffWriter.java

I edited my original post. Thanks to everyone for the comments, it helped me to look in the right direction.

Answer 2

In the SCIFIO project , we summarize the Bio-Formats image I / O structure for targeted scientific visualization in general, in addition to just microscopy and life sciences. Now the SCIFIO API is in beta, but includes TIFF , which can, of course, read and write in tiles. Feedback from the API and errors on the SCIFIO mailing list is always welcome!

Answer 3

Unlike your comment, it's actually not that difficult to implement your own TIFF writer.

The range is available for download here . In particular, on pages 13-14 (almost) everything you need to understand what TIFF is and how to write it down.

Please note that according to specification 6.0 (current as of September 2013), the maximum image size is 4 GB. I would suggest contacting BigTiff . The differences in specifications are listed in the second link.

Answer 4

You have a double array of short ones. Instead of storing it in memory, you need to save it to disk and manipulate it on disk. BufferedImage will not help. You need a library (or write a library) that allows you to manipulate the image on disk without the need to fully load the image into memory.

You do not want to access each value individually. Instead, you'll want to do something like this:

• read the block (maybe 4k, 8k or 40k, which makes sense).
• processes the entire block.
• write a block to disk.
• Go to step 1 to complete.