Open Microscopy Environment

BigTiff=true and LZW compression are there out of necessity, as our data are large.
(again, among tested computers only 3 exhibited that "slow setBytes" behaviour, for others these params were OK).

just tried single-frame bfsave test on fast machine as You suggested -

I=uint16(zeros(1191,2559,1,1,1));tic;bfsave(I,'dummy.OME.tiff');toc
Elapsed time is 0.240153 seconds.

will try this on slow comp later and be back, they are busy/not available at the moment..

Cheers,
Y.

...
Update - yes Your last suggestion was fruitful - "bfsave" saves a frame in a fast way (about same 0.3s) on the "slow" machine!

Now I try to compare the differences of my code from bfsave.

Many thanks, - it awakes a hope.. will be back.

Best,
Y.

Hi Yuriy,

it would be interesting to know whether the Compression/BigTiff optional parameters of bfsave (with the simple test case used previously) have any effect on performance on your "slow" system to start with.

Cheers,
Sebastien

Hi Sebastien,

As far as I can see for now, BigTiff, Compression, and presence of custom XML metadata have no effect.

Also I found the following - the function "test_add_plane.." sets up and uses global Bioformats "writer" (this is needed for LabView). It saves at ~11s per frame.
There is another function there that takes stack input and uses locally defined "writer" - it works at ~5s per frame. This is still slow - on "fast" machine they are both at ~0.3s.

Best,
Y.

Hi Yuriy,

what about the performance of the following piece of code on your slow machine

Code: Select all

if exist('dummy.ome.tiff', 'file'), delete('dummy.ome.tiff'); end
I=zeros(1191,2559,1,1,5, 'uint16');
tic;bfsave(I,'dummy.ome.tiff');toc


Best,
Sebastien

Thanks Sebastien, - it is still in "fast" category

>> if exist('dummy.ome.tiff','file'), delete('dummy.ome.tiff');end
>> I = zeros(1191,2559,1,1,5,'uint16');
>> tic;bfsave(I,'dummy.ome.tiff');toc
Elapsed time is 2.663927 seconds.

This is 0.5328s per frame.
However, for a "single" frame, it is 0.274s - roughly twice faster.
Here is the dependence on number of frames for bfsave (albeit in acquisition, we don't create memory block)
1 - 0.274s
5 - 0.5328
20 - 0.5801
40 - 0.6009
162 - 0.5954 (our data size)

this code

Code: Select all

       
acc = zeros(1,nPlanes);
       for index = 1 : nPlanes
            I = imread([folder filesep file_names{index}],extension);           
            t0 = tic;
            writer.saveBytes(index-1, getBytes(I));
            telapsed = toc(t0);
            acc(index)=telapsed;           
        end
writer.close();       
speed = mean(acc);


gives 5.74s per frame.
again, similar function where writer made "global", gives 11.1s p.f.
Mystery persists..

Best,
Y.

Hi Yuriy,

the factor 10 is surprising as I don't see anything wrong with your loop. Do you have the complete code that gives you 5.7s per frame (including writer initialization) rather than a code snippet?

Sebastien

yes certainly - the file is "save_OPT_stack_as_OMEtiff_with_metadata" in the bundle I've sent previously, and it is called by running "test_save_OPT_stack_as_OMEtiff_with_metadata" script.

(I keep this stuff on github at https://github.com/yalexand/Imperial-OMEkit)

Best,
Y.

...
differences seem to be the following -
bfsave deduces metadata from the input image data-block:

Code: Select all

metadata = createMinimalOMEXMLMetadata(I, ip.Results.dimensionOrder);

in the proc under question, this is replaced by something longer - and causes slowing down of the "saveBytes" - at least on some but quite critical Windows machines -

Code: Select all

       % Create metadata
        toInt = @(x) ome.xml.model.primitives.PositiveInteger(java.lang.Integer(x));
        OMEXMLService = loci.formats.services.OMEXMLServiceImpl();
        metadata = OMEXMLService.createOMEXMLMetadata();
        metadata.createRoot();
        metadata.setImageID('Image:0', 0);
        metadata.setPixelsID('Pixels:0', 0);
        metadata.setPixelsBinDataBigEndian(java.lang.Boolean.TRUE, 0, 0);

        % Set dimension order
        dimensionOrderEnumHandler = ome.xml.model.enums.handlers.DimensionOrderEnumHandler();
        dimensionOrder = dimensionOrderEnumHandler.getEnumeration('XYZCT');
        metadata.setPixelsDimensionOrder(dimensionOrder, 0);

        % Set pixels type
        pixelTypeEnumHandler = ome.xml.model.enums.handlers.PixelTypeEnumHandler();
        if strcmp(class(I), 'single')
            pixelsType = pixelTypeEnumHandler.getEnumeration('float');
        else
            pixelsType = pixelTypeEnumHandler.getEnumeration(class(I));
        end

        metadata.setPixelsType(pixelsType, 0);

        metadata.setPixelsSizeX(toInt(sizeX), 0);
        metadata.setPixelsSizeY(toInt(sizeY), 0);
        metadata.setPixelsSizeZ(toInt(sizeZ), 0);
        metadata.setPixelsSizeC(toInt(sizeC), 0);
        metadata.setPixelsSizeT(toInt(sizeT), 0);

        % Set channels ID and samples per pixel
        for i = 1: sizeC
            metadata.setChannelID(['Channel:0:' num2str(i-1)], 0, i-1);
            metadata.setChannelSamplesPerPixel(toInt(1), 0, i-1);
        end

hmmm.. something is forgotten in the second piece of code, but present in the first method?

Best,
Y.

Hi Yuriy,

First, many thanks for the repository link, I suggest we start working there using issues/Pull Requests so that we can pinpoint what goes wrong with the code. If your assumption is correct, substituting the long block in your function by something like

Code: Select all

metadata = createMinimalOMEXMLMetadata(repmat(I, 1,1,sizeZ, sizeC, sizeT));


should restore the performance assuming I is the first plane to be saved. Could you try this and let me know? Else I can open a PR against your repository tomorrow to test this?

Best,
Sebastien