We're Hiring!

Importing Performance

General user discussion about using the OMERO platform to its fullest. Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

Importing Performance

Postby 30laur » Mon Aug 16, 2010 5:08 am

Hi,
were doing some simple performance testing of OMERO using the importer tool. We have about 19MB to 20MB of 145 images and we noticed that it takes about 1 minute to go through the following OMERO importer process (we see these on the Importer status bar)
1. 1/3
2. 2/3
3. 3/3
4. Archiving
5. Thumbnail
6. Prepping
7. Updating DB

The most CPU intensive process is 7. Updating DB
The most longest process is 7. Updating DB
The second most longest process is 5. Thumbnail

Can anyone explain what the status above are doing and how we could reduce the CPU and timeframes?
e.g. are there any OMERO config parameters we could tune?
30laur
 
Posts: 5
Joined: Wed Jul 21, 2010 5:13 am

Re: Importing Performance

Postby jrswedlow » Mon Aug 16, 2010 6:21 am

Hi 30laur-

To answer your question, we need a bit more detail.

What file format are you uploading? What are the x,y sizes of the images you are loading?

Cheers,

Jason
User avatar
jrswedlow
Team Member
 
Posts: 38
Joined: Mon May 25, 2009 9:28 pm

Re: Importing Performance

Postby 30laur » Tue Aug 17, 2010 2:32 am

Hi Jason,
uploading NEF files which average around 19MB to 20MB in size.
Dimensions (XY) 4320 x 2868
Pixel Size (XY) 84.6677 x 84.667

We are currently "testing the performance of OMERO" as we wanted to see how it could handle a "big load".
Stats so far
NEF's-> 143/145 files uploaded, about 20 MB per file, 2.9 GB total files. 2 hours & 41 minutes
JPGs-> 1.13 hours, 482 JPG files of various sizes (300kb to 500kb) , 369.08 gb in total

Performance will be the biggest issue - i.e. retrieval of files etc...
Regards,
Raymond
30laur
 
Posts: 5
Joined: Wed Jul 21, 2010 5:13 am

Re: Importing Performance

Postby cxallan » Tue Aug 17, 2010 11:07 am

I assume these images are photographs?
cxallan
Site Admin
 
Posts: 509
Joined: Fri May 01, 2009 8:07 am

Re: Importing Performance

Postby 30laur » Tue Aug 17, 2010 10:29 pm

Yes
30laur
 
Posts: 5
Joined: Wed Jul 21, 2010 5:13 am

Re: Importing Performance

Postby cxallan » Thu Aug 19, 2010 5:55 pm

So first some background. We consider all images quantitative data and therefore are very careful to preserve the pixel intensities of those images. Thus in your case we completely decompress each of the JPEG and NEF files into their core 8-bit (or higher) colour components. This results in a huge explosion of image data volume. OMERO supports no compression internally at present.

Furthermore, OMERO is not a system designed with digital photography use cases in mind and is not going to give you good performance when working with these types of files. This is a result of our data storage mechanisms, outlined above, and our design decisions surrounding the use cases of digital microscopy.

Are their digital microscopy use cases you have in mind that you'd like to optimize for? What sort of hardware are you running the OMERO server and clients on?
cxallan
Site Admin
 
Posts: 509
Joined: Fri May 01, 2009 8:07 am


Return to User Discussion

Who is online

Users browsing this forum: Google [Bot] and 1 guest

cron