We're Hiring!

Large ModuloAlongT data - import is slow

General and open developer discussion about using OMERO APIs from C++, Java, Python, Matlab and more! Please new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

If you are having trouble with custom code, please provide a link to a public repository, ideally GitHub.

Large ModuloAlongT data - import is slow

Postby yuriy_alexandrov » Mon Dec 09, 2013 3:35 pm

Hi All,

Our FLIMfit software can work with non-imaging ("single-pixel") data.
When importing such data to Omero, the lifetime axis is chosen to be not XY - the convention is ModuloAlongT.

For such "single-pixel" data, I noticed that placing lifetimes along non-XY dimension is very slow.

In particular, we have real-world high-resolution 31 channel TCSPC data from cuvette system (X=1, Y=1, Z=1, T=4096, C=31).
Every 4096-bin channel is imported in 30 seconds, so it takes 16 minutes to import the whole 1.3M file.

My laptop is slow but I think the problem is still there, - for the comparison, the 32M (4-channel 128x128 256-bin) sdt image is imported in 30 seconds.

The slow part of the code is this, and the problem is most likely in "setPlane" function.
Code: Select all
           
...
bytear = ConvertClientToServer(pixels, plane) ;
rawPixelsStore.setPlane(bytear, int32(z-1), int32(c-1), int32(t-1));               
...

Is it possible to speed it up, say if applying other "set" function?

Thanks,
Y.
Last edited by yuriy_alexandrov on Fri Jan 10, 2014 4:42 pm, edited 2 times in total.
yuriy_alexandrov
 
Posts: 126
Joined: Thu Oct 25, 2012 2:06 pm

Re: Large ModuloAlongT data - import is slow

Postby sbesson » Tue Dec 10, 2013 11:58 am

Hi Yuriy,

Looking at the mat2omeroImage_native function in your code, here are 2 thoughts:
1- the API is not optimized for the type of data you are trying to upload. Here, you need to invoke setPlane for every single pixel in your input matrix within a triple loop.
2- profiling your code, a lot of the overhead actually arises from the setChannelGlobalMinMax method that is being invoked on every channel.
Typically,

Code: Select all
mat2omeroImage_native(session, zeros(1,1,1,1000,'uint16'), 'uint16', 'test', 'desc','')


runs in 392s with the MATLAB Profiler turned on where 40s are spent in ConvertClientToServer and setPlane while 298s are spent in setChannelGlobalMinMax.

I opened a PR against your code (https://github.com/openmicroscopy/Imper ... it/pull/65) suggesting refactoring changes. With this PR in, the same test function as above runs in the same condition in only 111s with 37s spent on the IUpdate.saveAndReturn function updating the channels. Can you try out these changes? This should at least decrease the total amount of time. I also think your function can be further optimised for this type of data.

Let us know if this works,
Sebastien
User avatar
sbesson
Team Member
 
Posts: 421
Joined: Tue Feb 28, 2012 7:20 pm

Re: Large ModuloAlongT data - import is slow

Postby i.munro » Tue Dec 10, 2013 1:33 pm

Thanks Sebastien. I'll merge that & get Yuriy to test when he gate back after Christmas.

Ian
i.munro
 
Posts: 50
Joined: Thu Apr 25, 2013 1:01 pm

Re: Large ModuloAlongT data - import is slow

Postby yuriy_alexandrov » Wed Dec 11, 2013 11:24 am

Thanks Sebastien,

I remember trying the similar simplified vesrion but didn't notice improvement.
I will look at it again in January.

Wish good Christmas and New Year to all!

Best regards,
Y.
yuriy_alexandrov
 
Posts: 126
Joined: Thu Oct 25, 2012 2:06 pm


Return to Developer Discussion

Who is online

Users browsing this forum: No registered users and 0 guests