We're Hiring!

slow retrieval of hypercube data from OMERO

General and open developer discussion about using OMERO APIs from C++, Java, Python, Matlab and more! Please new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

If you are having trouble with custom code, please provide a link to a public repository, ideally GitHub.

slow retrieval of hypercube data from OMERO

Postby gyellen » Mon Nov 17, 2014 2:00 pm

Hi all -
I've been wrestling with the slow retrieval of FLIM data (and more generally of all data) from an OMERO server.

I can test the actual transport time of data from the server by fetching the data of a large FileAnnotation. I can reliably read 1.2 MBytes in 0.15 - 0.32 sec (this is the time just for the store.read(size) operation).

In contrast, when these data are stored as 256 planes of 2 channels of 128x128 uint16 pixels (which is 1.68 MBytes uncompressed), it takes 2.25 - 2.5 seconds for the store.getHypercube(...) operation.

It even takes 0.6 - 0.7 sec to retrieve only 60 planes of data using getHypercube. I can't tell from reading the code whether the planes are assembled into a hypercube on the server side, or whether the planes are requested individually by the client. In any case, the performance is very slow.

I'm inclined at this point just to save my FLIM data as a FileAnnotation to the smaller amount of image data, but this defeats the interoperability of having data stored as OMERO images. (I would even consider storing ALL of my data as FileAnnotations!).

It would be better if the performance could be improved instead.

Best,
Gary
Last edited by gyellen on Mon Nov 17, 2014 4:43 pm, edited 1 time in total.
gyellen
 
Posts: 24
Joined: Fri Oct 24, 2014 1:19 am

Re: slow retrieval of hypercube data from OMERO

Postby gyellen » Mon Nov 17, 2014 4:42 pm

I misstated one of the benchmarks. The correct values are (using average values):

1.2 MByte file, retrieved by rawFileStore.read(size): 0.23 sec (min of 0.14 s)

The same data in this 1.2MB file, moved as a hypercube:
512x(128x128) uint16, retrieved by rawPixelsStore.getHypercube(offset,size,step): 2.38 sec (min of 2.23 s)

I also realized that the main difference might be data compression. The uncompressed size of the data is actually 16.7 MByte. So if the getHypercube data are being transported in uncompressed form, the difference in time would be explained.

Is there any data compression used during server-client transport of pixels data?

Thanks - gary
gyellen
 
Posts: 24
Joined: Fri Oct 24, 2014 1:19 am

Re: slow retrieval of hypercube data from OMERO

Postby gyellen » Mon Nov 17, 2014 5:47 pm

Aha! I added the following lines to the ice.config file:

Ice.Override.Compress=1
Ice.Compression.Level=8
# (doesn't improve with Level=9)

This reduces the speed of the already-compressed file transport (0.5 s, up from 0.23 s), but dramatically improves the speed of the getHypercube move (0.87 s, down from 2.23 s).

I still wonder if getHypercube performance can be improved, if the client is currently asking the server for individual planes (I can't tell). Would it be possible instead for the hypercube structure to be 'assembled' on the server side, and sent as a single message to the client?

Thanks - gary
gyellen
 
Posts: 24
Joined: Fri Oct 24, 2014 1:19 am

Re: slow retrieval of hypercube data from OMERO

Postby i.munro » Wed Nov 19, 2014 10:37 am

Hi again Gary

Sorry I can't help with your issue but I am interested in the performance increase you saw by modifying the config file. Do you think this approach might work with our uncompressed data?

Thanks

Ian
i.munro
 
Posts: 50
Joined: Thu Apr 25, 2013 1:01 pm

Re: slow retrieval of hypercube data from OMERO

Postby sbesson » Wed Nov 19, 2014 11:27 am

Hi Gary,

thanks for investigating this performance issue. Are you also using an unencrypted session as described in http://www.openmicroscopy.org/site/supp ... ure-client? If not, this may be another way for you to speed up data transfer.

Best,
Sebastien
User avatar
sbesson
Team Member
 
Posts: 421
Joined: Tue Feb 28, 2012 7:20 pm

Re: slow retrieval of hypercube data from OMERO

Postby gyellen » Wed Nov 19, 2014 11:34 am

Hi Ian -
Yes, I think it should help - the only time I saw a decrease in performance was when I moved data of an already-compressed OriginalFileI.

The trick is to include the lines I mentioned in the ice.config file, but also to make sure that the file is used in the loadOmero call. I know that including some/all arguments in the loadOmero call prevents ice.config from being loaded.

Sebastien - I'll have to check on the encryption selection - I'll get back to you.

Best,
Gary
gyellen
 
Posts: 24
Joined: Fri Oct 24, 2014 1:19 am

Re: slow retrieval of hypercube data from OMERO

Postby sbesson » Wed Nov 19, 2014 11:55 am

Regarding the connection via a configuration file, there are two ways to achieve it (assuming you have a valid configuration file)

- either pass no argument to loadOmero/connectOmero. In this case, the function will pick up the configuration file defined by the ICE_CONFIG environment variable or found in the MATLAB path
Code: Select all
[c,s] = loadOmero()

See https://github.com/openmicroscopy/openm ... mero.m#L49

- or pass the absolute path to the configuration file directly to loadOmero
Code: Select all
[c,s] = loadOmero('/path/to/ice.config')

See https://github.com/openmicroscopy/openm ... mero.m#L59

Best,
Sebastien
User avatar
sbesson
Team Member
 
Posts: 421
Joined: Tue Feb 28, 2012 7:20 pm

Re: slow retrieval of hypercube data from OMERO

Postby i.munro » Thu Nov 20, 2014 9:24 am

Thanks both. I'll experiment as soon as I get a chance.
Regards

Ian
i.munro
 
Posts: 50
Joined: Thu Apr 25, 2013 1:01 pm


Return to Developer Discussion

Who is online

Users browsing this forum: Google [Bot] and 0 guests