VSI Pyramidal File format

[sbesson]

Hi Oli,

So I've been playing around and no regression is making it any faster. So I deeply apologize for thinking the problem lies within BioFormats.

No worries at all. Regressions do tend to slip through cracks and flagging them early is welcome. Thanks for taking the time to check earlier versions.

The one called Fast gets parsed in 0.36 s and the thumbnail I need opens in 1.76 s (470x761 px)
The one called Slow gets parsed in 2.14 s and the thumbnail I need opens in 8.54 s (513x81 px)
The one called Very Slow gets parsed in 22.04 s and the thumbnail I need opens in 91.24 s (580x100 px)

Thanks for uploading the files and providing us the metrics. We can certainly reproduce similar variation factors against the same files locally.

So could it be that Olympus changed something in the way they store their files that makes the reading go through some other path that takes longer? Can yu uploaded, a notable difference we can see between the files is the Production Version, namely 2.4 for thou find a difference between the first and second files that would justify this 4x loss in speed?

Looking at the metadata for the files yoe Fast.vsi and 2.7 for the two slow files. Looking at our own representative VSI data using for testing, we only have datasets produced with Product Version 2.6 and under. So it all looks like the performance degradation is specific to files generated by a recent version of the Olympus Software.

We need more investigation to determine whether this could be due to a different file format, compression level etc In the meantime, could you confirm the metadata and the pixel data read by Bio-Formats is actually correct albeit slow?

Best,
Sebastien

[Oli]

Dear Sebastien,

Thank you for the support. I can confirm that the pixel data is loaded properly, albeit slowly. Metadata seems OK as well (We use the "Microscope Device Model #1"Metadata field to check whether it's a color or BW camera).

Best

Oli

[dgault]

Hi Oli,

I have been able to spend more time analysing the timings of the file reader today. It appears that the time delay is being spent attempting to read a tag value within the file which we do not recognise. I have opened a code PR https://github.com/openmicroscopy/bioformats/pull/2365 to skip attempting to read any values which we do not support. This should reduce the load times of the files without affecting any of the existing metadata.

With Thanks,
David