We're Hiring!

IMPORT planes in parallel threads, ims format

Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats
Please note:
Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats

If you are having trouble with image files, there is information about reporting bugs in the Bio-Formats documentation. Please send us the data and let us know what version of Bio-Formats you are using. For issues with your code, please provide a link to a public repository, ideally GitHub.

IMPORT planes in parallel threads, ims format

Postby romainGuiet » Wed Sep 27, 2017 7:30 am

Dear OME community,

I’m currently testing teraSticher (https://github.com/abria/TeraStitcher/wiki) that proposes to export files as :
-ims
-hdf5
Unfortunately, I can’t use (yet?) the h5 file with Fiji/bio-formats, BUT I can open the ims file.

I wrote an ImageJ plugin that can process planes (Z, C, T, Series) on parallel threads, thanks to the Bioformats virtualStack capabilities. (the processing get so fast when you have 32-cores at your disposal!).

While it works fine (multiple planes in parallel simultaneously) if the file is an ICS file,
When it’s an IMS file, the task is done on multiple cores BUT sequentially! It seems that the processing of each plane is done on a different core but starts when the previous one is done!

Would you have any input on this issue?
Is that a limitation of the .ims file format ? or the way the reader access data in the .ims file ?

Thank you for your help,

Romain
romainGuiet
 
Posts: 5
Joined: Wed May 10, 2017 12:30 pm

Re: IMPORT planes in parallel threads, ims format

Postby dgault » Wed Sep 27, 2017 3:13 pm

Hi Romain,

Although we do not guarantee that Bio-Formats readers are fully thread safe my initial impression is that there should be no particular difference between the ICS and IMS image readers. However I have not been able to test the IMS reader specifically so there may indeed be a limitation. Would you have a sample macro or code snippet of the type of processing you are attempting?

David Gault
User avatar
dgault
Team Member
 
Posts: 208
Joined: Fri Aug 14, 2015 2:56 pm

Re: IMPORT planes in parallel threads, ims format

Postby romainGuiet » Mon Oct 02, 2017 9:01 am

Hi David,
Thank you for your reply.
You will find here a zip containing the plugin and the source.

The multithread part is based/copied of : “Multithreaded programming for ImageJ” is in the IMS_issue_4OME.java
The image initialization is also based on an example and can be found in the IMS_issue_4OMEParam.java

To test it, you can use this dataset, an image 20Zx3C in formats ids/ics and ims.

Using the ids, the planes are processed in parallel simultaneously ( in about 20s) while using the ims the planes are processed in parallel “sequentially” ( takes about 180s).

Thank you for your help and suggestions to solve this.
Romain
romainGuiet
 
Posts: 5
Joined: Wed May 10, 2017 12:30 pm

Re: IMPORT planes in parallel threads, ims format

Postby dgault » Tue Oct 03, 2017 12:00 pm

Thanks Romain, I have downloaded the dataset and source provided. Over the next day or two I will test and debug the processing to see if I can isolate any improvements or bugs. I shall keep this thread up to date with any progress I make.
User avatar
dgault
Team Member
 
Posts: 208
Joined: Fri Aug 14, 2015 2:56 pm

Re: IMPORT planes in parallel threads, ims format

Postby dgault » Thu Oct 05, 2017 2:15 pm

To follow up on this, I have been able to take a look through the source code in more detail and in general the structure of the plugin looks good from a Bio-Formats perspective.

The only section in which Bio-Formats will synchronise the threads should be the getProcessor call on BFVirtualStack. This may be the cause of the apparent sequential execution, although I am still unsure as to why IMS images appear to be affected specifically. My suggestion would be to test retrieving the processors earlier outside of the threaded execution.
User avatar
dgault
Team Member
 
Posts: 208
Joined: Fri Aug 14, 2015 2:56 pm

Re: IMPORT planes in parallel threads, ims format

Postby romainGuiet » Wed Oct 11, 2017 7:58 am

>To follow up on this, I have been able to take a look through the source code in more detail and in general the structure of the plugin looks good from a Bio-Formats perspective.

I'm glad that "the structure of the plugin looks good"

>The only section in which Bio-Formats will synchronise the threads should be the getProcessor call on BFVirtualStack. This may be the cause of the apparent sequential execution, although I am still unsure as to why IMS images appear to be affected specifically.
That's my main concern here, why IMS is treated differentially than IDS?

> My suggestion would be to test retrieving the processors earlier outside of the threaded execution.
Could you please explain how it would help us ?
Correct me but then it won't be parallel anymore ? so we won’t see any difference?
OR
we'll have to pre-load the whole data set and there is no use for virtual stack and we can't process files bigger than the RAM.

Thanks for your help with this,
Romain
romainGuiet
 
Posts: 5
Joined: Wed May 10, 2017 12:30 pm

Re: IMPORT planes in parallel threads, ims format

Postby dgault » Thu Oct 12, 2017 8:11 am

Hi Romain,

I have pulled out the core parts of the code from the plugin to a standalone Java program for testing with Imaris files. From what I am seeing the main threaded section is being executed in parallel as expected. Surrounding it with timing commands is also showing an improvement in execution time with a greater number of threads. This appears to confirm that the issue is not fundamentally linked to Imaris files.

As mentioned previously the getProcessor call on BFVirtualStack is a synchronised method, from debugging and checking execution times it does seem that some threads are being held up at this point waiting for the getProcessor call. The effect of this will vary depending on the particular image. For some smaller sample images this does not have much of a negative effect. From the sample file you provided however this getProcessor call is taking a significant portion of the execution time which causes it to hold up other threads execution giving the impression that the processing is happening sequentially.

David Gault
User avatar
dgault
Team Member
 
Posts: 208
Joined: Fri Aug 14, 2015 2:56 pm

Re: IMPORT planes in parallel threads, ims format

Postby Oli » Wed Oct 18, 2017 2:36 pm

Dear David,

Thank you so much for your time and effort!

From what you are saying, I am interpreting it this way
This getProcessor() from BFVirtualStack call ends up being slower for .ims files than for the .ids/.ics, even though we call for the same pixel data.

It turns out it's so slow that it becomes the main bottleneck of this (very simple) processing, making it seem like it's not running in parallel…

Is this correct?

Do you have any thoughts on how we can change the structure of the code to try and circumvent this issue? Perhaps declare multiple readers to read the same file at different locations? Something more elegant?

As always, thank you very much for your time.

Oli
Oli
 
Posts: 71
Joined: Mon Nov 29, 2010 4:57 pm

Re: IMPORT planes in parallel threads, ims format

Postby dgault » Thu Oct 19, 2017 10:07 am

Hi Oli,

That is correct, it is the call to getProcessor that is taking much longer for the ims files and causing the bottleneck here. In terms of improving the performance my initial thoughts were the same as yours, to split the getProcessor over an array of BFVirtualStack objects.

I ran some initial tests for this and using an array of BFVirtualStack with the same reader allowed the threads to run in parallel although it looks to have thrown IllegalStateExceptions in the NetCDF library. Running with an array of BFVirtualStack each with a separate reader appears to have worked much better with a significant speed up in execution time. This will come at the cost of extra overhead of memory performance though, for this particular file it accounted for approx 20Mb for each extra BFVirtualStack and 20Mb per processor.

David Gault
User avatar
dgault
Team Member
 
Posts: 208
Joined: Fri Aug 14, 2015 2:56 pm

Re: IMPORT planes in parallel threads, ims format

Postby Oli » Fri Feb 02, 2018 10:21 am

Hi David,

Sorry for the late reply. We checked and indeed it does work better!

So we would initialize as many readers and BFVirtualStack as needed, then split the threads.

Thanks for checking the extra overhead, which is perfectly acceptable in our case.

Thanks again for all the great work!

Oli
Oli
 
Posts: 71
Joined: Mon Nov 29, 2010 4:57 pm


Return to User Discussion [Legacy]

Who is online

Users browsing this forum: No registered users and 0 guests