Bioformats OME.tiff saving - performance

[yuriy_alexandrov]

Thanks Sebastien,

As far as only I have access to "slow" machines, - I will test Your suggestion tomorrow morning, ASAP.

On github rep - I added You to collaborators list, so the direct change should be allowed.
If something isn't set well - could we please discuss details via usual e-mail (my knowledge of PR procedure is mmm... quite rudimentary ). The repo might be useful for our possible next app intended for OPT.

Best,
Y.

[yuriy_alexandrov]

Hi again Sebastien,

It starts to clarify, bit by bit.
I checked the metadata initialization method You suggested, which is according to repmat syntax

Code: Select all

metadata = createMinimalOMEXMLMetadata(repmat(I, [1 1 sizeZ sizeC sizeT]));


Now attention please.
When this way of metadata initialization is used, - the "Compression" setting influences performance.
Commenting out the line

Code: Select all

writer.setCompression('LZW');

- brings saving to fast category, ~0.6 secs pf versus ~6 secs when this line is uncommented.

So I will switch to this metadata initialization mode everywhere.
There is however some concern about how "repmat" will behave for big data.

What is Your opinion - is it worth looking in Bioformats, maybe to make it more, mm.. consistent?
Data compression is always desirable, but for now, one needs to check every other computer on the matter of possible OME.tiff writing slowing down.

Many thanks for help.

Best,
Y.

[sbesson]

Hi Yuriy,

for what you want to do `repmat` is certainly not a viable long-term solution. It was mostly for debugging. Can we confirm the issue comes down to the compression for multi-plane images? Is this is the case, the following code should exhibit slow performance on your machine:

I = zeros(1191,2559,1,1,5,'uint16');
tic; bfsave(I, 'dummy.ome.tiff', 'Compression', 'LZW');toc

As for `bfsave/createMinimalXMLMetadata`, your concerns are well-founded. You should keep in mind this function was mostly written as a template/starting point for small images. It certainly does not work out-of-the-box for 1- very large images, 2- screens, 3- images with extended metadata. Without putting too much thought into, it, a quick workaround in your specific case could be something like:

metadata = createMinimalOMEXMLMetadata(I);
metadata.setPixelsSizeZ(toInt(sizeZ), 0);

In general, the extension of these functions is largely driven by the needs of individual projects like yours so I am happy to consider any suggestion for improving them in a more generic way. What are your thoughts about:

metadata = createMinimalOMEXMLMetadata(I, 'sizeZ', sizeZ, 'sizeC', sizeC. 'sizeT', sizeT);

where I would be a single plane (used for XY dimensions and the pixel type) but where parameters could override the ZCT dimensions.

Best,
Sebastien

[yuriy_alexandrov]

Yes certainly such constructor-function would be very handy.
I will check Your suggestions now.

Cheers,
Y.

[yuriy_alexandrov]

yes Your'e right - here are the outputs (both performances are low but comp is accessed remotely so it might do something else in parallel - the order of magnitude difference is still there)

>> tic; bfsave(I, 'dummy.ome.tiff', 'Compression', 'LZW');toc
Elapsed time is 27.071436 seconds.
>> tic; bfsave(I, 'dummy.ome.tiff');toc
Elapsed time is 2.898138 seconds.

Your second suggestion has worked too - with that small change in ini, it is still in fast mode.
So by this I believe we'll have the fix to the problem!

And what with compression? - if say one needs it badly, and speed at the same time, but checks the machine and realizes oops it's bad luck, it can't provide both.. Maybe I'm asking too many questions.

Anyway, with that great and gracious help from You, we can slow down on this thread..

Best wishes,
Y.

[mlinkert]

Hi Yuriy,

And what with compression? - if say one needs it badly, and speed at the same time, but checks the machine and realizes oops it's bad luck, it can't provide both.. Maybe I'm asking too many questions.

Compression will always have a performance overhead, so writing with compression will by necessity always be slower than writing without compression. If fast write speed is needed for acquisition, your best bet would be write without compression first, and then convert to a compressed file later (when speed is less critical).

-Melissa

[yuriy_alexandrov]

thanks Melissa - on compression, You confirmed my vague suspicions , it was a surprise that on some machines slowing down is almost unnoticeable, on others it is order of magnitude.

Sebastien - could You please check the function "createMinimalOMEXMLMetadata" - I think there is bug on the lines -

Code: Select all

% sizeX = size(I, 2);
% sizeY = size(I, 1);


X and Y sizes are flip-flopped.
I've put the fix on my github dir, temporarily. At least, it allows working without additional XY transposes.

Best,
Y.

[sbesson]

Hi Yuriy,

The first two dimensions of the matrix input for bfsave and createMinimalOMEXMLMetadata are expected to be MxN where M is the size along the Y axis and N the size along the X axis. This is a convention in agreement with the way MATLAB deals with images (see e.g. the output of imread).
Is the fix in your code related to the fact your initial plane is transposed in line https://github.com/yalexand/Imperial-OM ... af30857L35?

Sebastien

[yuriy_alexandrov]

I removed all transposes form these two procedures and pushed changes just now.
There are no (I') anymore.
To keep images not scrambled, I needed to revert XY sizes in "createMinimalOMEXMLMetadata".

To keep createMinimalOMEXMLMetadata as is - one needs to use I' instead of I everywhere AFAIR/U.

The code as it is now, saves OME-tiff that one can import to Omero and ensure it is fish there but not a mishmash (so I'm happy ). Test data are in the folder, too, maybe You can check it when have minute.

Best,
Y.

[yuriy_alexandrov]

oh I see what You mean - when I look at images in Matlab, they look transposed if compared to Omero-Insight.
Do we presume that Insight displays images as they are opened in Matlab?
mm...

Will look at it in Monday, hopefully there is something trivial here even if it needs fix.

Have good week-end,
Y.