Bioformats OME.tiff saving - performance

[yuriy_alexandrov]

BigTiff=true and LZW compression are there out of necessity, as our data are large.
(again, among tested computers only 3 exhibited that "slow setBytes" behaviour, for others these params were OK).

just tried single-frame bfsave test on fast machine as You suggested -

I=uint16(zeros(1191,2559,1,1,1));tic;bfsave(I,'dummy.OME.tiff');toc
Elapsed time is 0.240153 seconds.

will try this on slow comp later and be back, they are busy/not available at the moment..

Cheers,
Y.

[yuriy_alexandrov]

...
Update - yes Your last suggestion was fruitful - "bfsave" saves a frame in a fast way (about same 0.3s) on the "slow" machine!

Now I try to compare the differences of my code from bfsave.

Many thanks, - it awakes a hope.. will be back.

Best,
Y.

[sbesson]

Hi Yuriy,

it would be interesting to know whether the Compression/BigTiff optional parameters of bfsave (with the simple test case used previously) have any effect on performance on your "slow" system to start with.

Cheers,
Sebastien

[yuriy_alexandrov]

Hi Sebastien,

As far as I can see for now, BigTiff, Compression, and presence of custom XML metadata have no effect.

Also I found the following - the function "test_add_plane.." sets up and uses global Bioformats "writer" (this is needed for LabView). It saves at ~11s per frame.
There is another function there that takes stack input and uses locally defined "writer" - it works at ~5s per frame. This is still slow - on "fast" machine they are both at ~0.3s.

Best,
Y.

[sbesson]

Hi Yuriy,

what about the performance of the following piece of code on your slow machine

Code: Select all

if exist('dummy.ome.tiff', 'file'), delete('dummy.ome.tiff'); end
I=zeros(1191,2559,1,1,5, 'uint16');
tic;bfsave(I,'dummy.ome.tiff');toc


Best,
Sebastien

[yuriy_alexandrov]

Thanks Sebastien, - it is still in "fast" category

>> if exist('dummy.ome.tiff','file'), delete('dummy.ome.tiff');end
>> I = zeros(1191,2559,1,1,5,'uint16');
>> tic;bfsave(I,'dummy.ome.tiff');toc
Elapsed time is 2.663927 seconds.

This is 0.5328s per frame.
However, for a "single" frame, it is 0.274s - roughly twice faster.
Here is the dependence on number of frames for bfsave (albeit in acquisition, we don't create memory block)
1 - 0.274s
5 - 0.5328
20 - 0.5801
40 - 0.6009
162 - 0.5954 (our data size)

this code

Code: Select all

       
acc = zeros(1,nPlanes);
       for index = 1 : nPlanes
            I = imread([folder filesep file_names{index}],extension);           
            t0 = tic;
            writer.saveBytes(index-1, getBytes(I));
            telapsed = toc(t0);
            acc(index)=telapsed;           
        end
writer.close();       
speed = mean(acc);


gives 5.74s per frame.
again, similar function where writer made "global", gives 11.1s p.f.
Mystery persists..

Best,
Y.

[sbesson]

Hi Yuriy,

the factor 10 is surprising as I don't see anything wrong with your loop. Do you have the complete code that gives you 5.7s per frame (including writer initialization) rather than a code snippet?

Sebastien

[yuriy_alexandrov]

yes certainly - the file is "save_OPT_stack_as_OMEtiff_with_metadata" in the bundle I've sent previously, and it is called by running "test_save_OPT_stack_as_OMEtiff_with_metadata" script.

(I keep this stuff on github at https://github.com/yalexand/Imperial-OMEkit)

Best,
Y.

[yuriy_alexandrov]

...
differences seem to be the following -
bfsave deduces metadata from the input image data-block:

Code: Select all

metadata = createMinimalOMEXMLMetadata(I, ip.Results.dimensionOrder);

in the proc under question, this is replaced by something longer - and causes slowing down of the "saveBytes" - at least on some but quite critical Windows machines -

Code: Select all

       % Create metadata
        toInt = @(x) ome.xml.model.primitives.PositiveInteger(java.lang.Integer(x));
        OMEXMLService = loci.formats.services.OMEXMLServiceImpl();
        metadata = OMEXMLService.createOMEXMLMetadata();
        metadata.createRoot();
        metadata.setImageID('Image:0', 0);
        metadata.setPixelsID('Pixels:0', 0);
        metadata.setPixelsBinDataBigEndian(java.lang.Boolean.TRUE, 0, 0);

        % Set dimension order
        dimensionOrderEnumHandler = ome.xml.model.enums.handlers.DimensionOrderEnumHandler();
        dimensionOrder = dimensionOrderEnumHandler.getEnumeration('XYZCT');
        metadata.setPixelsDimensionOrder(dimensionOrder, 0);

        % Set pixels type
        pixelTypeEnumHandler = ome.xml.model.enums.handlers.PixelTypeEnumHandler();
        if strcmp(class(I), 'single')
            pixelsType = pixelTypeEnumHandler.getEnumeration('float');
        else
            pixelsType = pixelTypeEnumHandler.getEnumeration(class(I));
        end

        metadata.setPixelsType(pixelsType, 0);

        metadata.setPixelsSizeX(toInt(sizeX), 0);
        metadata.setPixelsSizeY(toInt(sizeY), 0);
        metadata.setPixelsSizeZ(toInt(sizeZ), 0);
        metadata.setPixelsSizeC(toInt(sizeC), 0);
        metadata.setPixelsSizeT(toInt(sizeT), 0);

        % Set channels ID and samples per pixel
        for i = 1: sizeC
            metadata.setChannelID(['Channel:0:' num2str(i-1)], 0, i-1);
            metadata.setChannelSamplesPerPixel(toInt(1), 0, i-1);
        end

hmmm.. something is forgotten in the second piece of code, but present in the first method?

Best,
Y.

[sbesson]

Hi Yuriy,

First, many thanks for the repository link, I suggest we start working there using issues/Pull Requests so that we can pinpoint what goes wrong with the code. If your assumption is correct, substituting the long block in your function by something like

Code: Select all

metadata = createMinimalOMEXMLMetadata(repmat(I, 1,1,sizeZ, sizeC, sizeT));


should restore the performance assuming I is the first plane to be saved. Could you try this and let me know? Else I can open a PR against your repository tomorrow to test this?

Best,
Sebastien