Is there a way to open JPEG2000 files greater than 2Gig?

[brad.polant]

I have seen this in some C-versions of it , but the class provided with BioFormats for reading JPEG2000 seems unhappy for larger files.
Thoughts?

[mlinkert]

You won't be able to open 2GB of pixel data as a single array (this is a limitation of Java, not Bio-Formats), but opening a smaller tile from an image with >2GB of pixel data is expected to work. If you are using Bio-Formats as a library, this means specifying the 'x', 'y', 'width', and 'height' parameters in the 'openBytes' method of IFormatReader; in ImageJ, you would need to choose the "Crop on import" option.

If opening a smaller tile does not work, then could you please let us know which version of Bio-Formats you are using and what the complete error message is?

[brad.polant]

The version is
5.02



The error is
loci.formats.FormatException: Buffer too small (got 200000, expected 786432).
at loci.formats.FormatTools.checkBufferSize(FormatTools.java:812)
at loci.formats.FormatTools.checkPlaneParameters(FormatTools.java:768)
at loci.formats.in.JPEG2000Reader.openBytes(JPEG2000Reader.java:161)
at com.corista.nis.slideReaders.JP2KSlideReader.main(JP2KSlideReader.java:68)

and this was the test code
JPEG2000Reader n = new JPEG2000Reader();
n.setId("O:\\BioImagene\\DH_2_10_200904031056.jp2");
int y =n.getSeriesCount();
byte[] t2 = new byte[200000];
byte[] tl = n.openBytes(0,t2, 0, 0, 512, 512);

what do you think?

[brad.polant]

Edited to add
This seems linked to an old OpenJDK bug
https://bugs.openjdk.java.net/browse/JDK-5101502


This is a more accurate stacktrace (right at at moment of exception) using the 5.10 develop branch
PixelInterleavedSampleModel(SampleModel).<init>(int, int, int, int) line: not available
PixelInterleavedSampleModel(ComponentSampleModel).<init>(int, int, int, int, int, int[]) line: not available
PixelInterleavedSampleModel.<init>(int, int, int, int, int, int[]) line: not available
PixelInterleavedSampleModel.createCompatibleSampleModel(int, int) line: not available
J2KReadState.readAsRaster() line: 429
J2KImageReader.readRaster(int, ImageReadParam) line: 556
JAIIIOServiceImpl.readRaster(InputStream, JPEG2000CodecOptions) line: 174
JPEG2000Codec.decompress(byte[], CodecOptions) line: 281
JPEG2000Codec.decompress(RandomAccessInputStream, CodecOptions) line: 253
JPEG2000Reader.openBytes(int, byte[], int, int, int, int) line: 181
JP2KSlideReader.main(String[]) line: 68

[mlinkert]

The stack trace and code indicate that the buffer being passed to openBytes is not correctly sized; the buffer must be at least as large as the number of bytes in the requested tile. The easiest way to calculate the minimum buffer size is:

Code: Select all

FormatTools.getPlaneSize(n, 512, 512)


assuming you are reading a 512x512 tile. If you still see an exception after increasing the buffer size, please let us know.

[brad.polant]

Thanks for getting back to me, I know you are busy.

The error message when you expand the buffer to a spacious size is

"java.lang.IllegalArgumentException: Dimensions (width=71376 height=41104) are too large"
and the line of code says to the effect of
if (w *x)>INT_MAX than assert.
even after FormatTools.getPlaneSize(n, 512, 512) tells us we need 764K bytes and we provide much more.

Does that make sense?

Edited to add
The jp2 files I am decoding is 345 Megs on disk, if I try to read a 20 meg one or so there is no issue. It seems to be trying to read *everything* in at once.

[mlinkert]

In that case, could you please upload the problematic file to http://qa.openmicroscopy.org.uk/qa/upload/ so that we can investigate further?

[mlinkert]

Thank you for uploading a sample file. We have work in progress to support reading tiles from large JPEG-2000 files:

https://github.com/openmicroscopy/bioformats/pull/1309

Once that shows as being merged, the file should be readable using the "Latest build" from http://downloads.openmicroscopy.org/bio-formats/5.0.3/.

[brad.polant]

Hi, I grabbed your changes and applied them to the latest code and this is what I found using showinf as a testing frame work.

I can read past any X coordinate, but on multiple flies there is a hard stop on reading past 4096 on the y axis.
It is not jagged, and it does not matter how big X is (i.e a 40000 ,8096 as opposed to a 10000 , 8096)
if will always fail at 4096 and will never break in the middle of a line.

This location is an absolute as well, if you start your read at 0,3072 you will only be able to read 1024 lines of data and if you start your read at 0,0 you will get 4096 lines of data! It's like a force field!
so to be clear the cut off is not related to the size of the offset requested only that after (N, 4096) there will not be valid data.

Thoughts?
I suspect the issue is in JAIIIOServiceImpl.java in readRaster, but I can't be sure

[mlinkert]

This was also mentioned on the pull request, and discussed briefly in IRC. To summarize:

https://github.com/openmicroscopy/biofo ... t-55352526

I don't have any further immediate ideas of what's going wrong, but I anticipate that we'll be able to investigate further next week.