OME-TIFF Export geometrically slows down - Potential fix

[a.herbert]

Hi,

I have been looking at the code that exports images as OME-TIFF. I noticed that for larger images the export seems to geometrically slow down. I decided to investigate this by adding some export timings to the Blitz log for the components/blitz/src/ome.services.blitz.impl.ExporterI class within the do_tiff(...) method.

I did a lot of reading of the code in the process and noticed that the sequential flag was not set on the BioFormats ImageWriter. This means that each time a new plane is written to the TIFF file, BioFormats reads the entire file to locate the positions of all the current IFD entries (see method loci.formats.tiff.TiffSaver.writeImageIFD(...)). If the current plane being written is less than the total count of IFD entries then the correct IFD entry is extracted for use.

Since the planes are written using incremental plane numbers I do not think this condition will ever be met. Thus it is safe to set the sequential flag to true.

I have tested this by adding the following code to ome.services.blitz.impl.ExporterI.do_Tiff(...):

Code: Select all

writer = new ImageWriter();
writer.setMetadataRetrieve(retrieve);
writer.setWriteSequentially(true);              // New code
writer.setId(file.getAbsolutePath());


Note: The call to setWriteSequentially(...) must be made before the call to setId(...) because that method initialises the TiffSaver object with the current sequential flag value.

Here are some timings generated using some artificial images of set sizes (timings are in seconds):

Code: Select all

Size (MB)   Off      On      Speed-up (Off / On)
0.5         0.306    0.323   0.948
1           0.133    0.272   0.488
2           0.529    0.672   0.786
4           0.654    0.636   1.030
8           1.290    1.217   1.060
16          2.624    2.357   1.113
32          5.776    4.517   1.279
64          13.620   9.049   1.505
128         36.041   18.024  2.000
256         106.802  35.596  3.000
512         354.905  71.353  4.974


As you can see the flag makes the export times scale linearly instead of geometrically. There is a significant 5 fold difference to a 500MB file. I expect the performance gains to be even better for larger files.

I have verified that the exported images render exactly the same in ImageJ for my test images and also some valid microscope images imported from DeltaVision files.

Can you verifiy if there is any reason why this flag should not be set? I assume you have a set of test cases that you can run to validate export.

I am using OMERO 4.3.0. If this fix has already been done as part of the work on exporting TIFFs larger than 4GB then at least I learned something about the code.

Regards,

Alex

[jmoore]

Thanks for this, Alex! I've created ticket 6701 and CC'd you on it. We'll evaluate and try our best to make sure this isn't outstanding for the upcoming 4.3.2.

Cheers,
~Josh.

[a.herbert]

Hi Josh,

My investigations into this have moved on a little bit. I am happy that setting sequential writing to true is OK. I then added some timings within the read-plane/write-plane loop to see how long the system was taking to do each task. The results are as follows (timing in seconds):

Code: Select all

MB    Total    Open     Save     
4     0.656    0.269    0.327   
8     1.220    0.474    0.665   
16    2.297    0.894    1.314   
32    4.546    1.756    2.609   
64    8.996    3.444    5.207   
128   17.964   6.809    10.714   
256   35.236   13.785   20.719   
512   71.011   27.257   42.306   


So you can see that BioFormats was taking longer to write the new TIFF file than OMERO takes to load the pixels.

To check to see if the latest version of BioFormats could do better than the one bundled with OMERO 4.3.0 I got the latest LOCI build and ran that:

Code: Select all

MB    Total    Open     Save     BF Speed-up
4     0.507    0.261    0.215    1.520
8     0.951    0.475    0.426    1.561
16    1.810    0.901    0.832    1.579
32    3.516    1.722    1.668    1.564
64    6.939    3.424    3.302    1.577
128   14.019   6.752    6.901    1.553
256   27.287   13.395   13.350   1.552
512   55.378   27.043   27.276   1.551


Here you see that the newer BioFormats is 50% faster. Good news and an easy fix.

It is now apparent that the read and write operations take about the same amount of time. So I thought about using a bit of threading to help out. I implemented a simple Runnable wrapper for the BioFormats ImageWriter. The object takes in the successfully initialised ImageWriter object that is created within the ExporterI.do_tiff() method. It has a saveBytes() method that adds byte[] arrays to a BlockingQueue. The Runnable's run() method takes the arrays off the queue and writes them using the ImageWriter. This allows the image to be saved in a separate thread. Timings are as follows:

Code: Select all

MB    Total    Open     Save     Total Speed-up
4     0.332    0.277    0.001    1.526
8     0.549    0.483    0.002    1.731
16    1.009    0.920    0.004    1.794
32    1.951    1.776    0.006    1.802
64    3.792    3.488    0.012    1.830
128   7.764    7.131    0.130    1.806
256   14.826   13.935   0.201    1.840
512   29.397   27.235   1.033    1.884


Note that the timings for the Save method are very low since this just times the addition of the byte[] array to the BlockingQueue. The total run-time for large files gains a significant speed-up.

So following on my my initial post I can now export a 512MB file in 29 seconds compared to the original 354 seconds; This is a speed-up of 12x.

I attach my test code for your reference. I know that the code runs cleanly when no problems occur. I have thrown exceptions from various code points to check that it cleans up correctly under error. However you can never be sure about the error handling for all the edge cases. Anyway I thought you might be interested in the raw timings. You may see it as beneficial enough to write a better version that integrates with your Executor and the threading environment for the Blitz server.

If you do not go for background writing then just setting the sequential flag to true and using the latest BioFormats is still a large improvement.

Regards,

Alex

[jmoore]

Alex, you'll probably have seen that your patch for calling "setWriteSequentially" has been incorporated for 4.3.2 as ticket 6701. I've also opened 6746 for the mult-threading, and CC'd you. Thanks again for all your help!

~Josh.

[a.herbert]

Hi,

This is an update to the work I have been doing on multi-threaded Tiff export. Since OMERO 4.3.2 I have found that the patch is not really needed.

In my original work on version 4.3.0 I imported some large Tiff files for testing. These were converted to image pyramids by the server. Thus when I split the reading of the pixel data and the writing of the big tiff using BioFormats I got a significant speed increase. I assume this is because reading a pyramid is hard work.

I applied the same code to 4.3.2 and imported some new test images. However these were not converted to pyramids (I presume due to the newer server rules applied to images regarding pyramids). When I did my testing I found that reading the pixel data took approximately 2% of the time it took to write the Tiff. This makes sense since within the OMERO code I see that this uses Java to tie the memory byte array for each plane to the raw byte file on disk. Getting the raw bytes is therefore very fast.

In my multi-threaded implementation I have to allocate a new array for each plane (instead of recycling the same byte memory block) and put them in a BlockingQueue. This adds about 25% extra runtime for each plane that is read. The planes can then be dequeued in a separate thread and written using BioFormats. Once all the planes have been read the main thread must wait for the background thread to finish writing all the data.

Overall the multithreading makes the system 2-3% faster on large files. This is something I do not think is worth worrying about. Here is the summary data for single/multi-threaded export. Timings are in milliseconds using an average of at least 3 exports for each data point:

Code: Select all

MB     Single      Multi       Ratio
0.5    46.03       44.42       0.97
1      79.78       79.11       0.99
2      112.59      106.21      0.94
4      217.53      210.07      0.97
8      436.03      421.24      0.97
16     864.59      840.96      0.97
32     1721.95     1675.20     0.97
64     3441.70     3361.51     0.98
128    6879.39     6727.62     0.98
256    13650.14    13358.87    0.98


I attach a spreadsheet with further information. The sheet contains the timings for opening the raw pixels, saving the pixels to the Tiff file and the amount of time the main thread had to wait for the background thread to finish writing the Tiff after all the planes had been read. The number of samples (N) is provided.

As I noted above this modification will improve performance when exporting pyramids. However with the new pyramid rules within v.4.3.2 I do not think it will be possible to have a pyramid that can be exported as a Big Tiff. The export code explicitly checks if the pixels are for a large image and errors if the image is too large for export. Thus I think that multi-threading the export is not needed given my testing.

Regards,

Alex