We're Hiring!

Moving large project to another group

General user discussion about using the OMERO platform to its fullest. Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

Re: Moving large project to another group

Postby jmoore » Thu Feb 06, 2014 7:25 am

Hi Tristan,

I'm currently repeating your experiment with 100 copies of the file you linked to. One question I have: did the moves you attempted come directly after the import? Since these TIFFs are fairly large (6K x 6K), the first thing the server does is to generate pixel pyramids for them at multiple resolutions. This processing overhead is certainly a portion of the CPU consumption you were seeing. Once the processing is done for all 100 images, I'll test the move operations and see how it goes.

Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Moving large project to another group

Postby jmoore » Thu Feb 06, 2014 5:13 pm

Hi again, Tristan,

having let the background processing finish, a move of 100 images in a single dataset on my local system takes 3-5 minutes. My expectation then is that the background work was causing much of the issues you saw. If you see any other notably slow actions, please let us know.

Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Moving large project to another group

Postby Sethur » Thu Feb 06, 2014 8:40 pm

Hi Josh,

I'm not entirely sure any more how much time I gave the system after the import before I started the move but it is very possible that I did it right away. I will repeat this experiment tomorrow.

On this note: Can you give me a recommendation on the memory settings for the different OMERO.server processes. Currently, I set Blitz to 8 GB and Indexer as well as PixelData to 1 GB. Our total system memory is 16 GB. Do you think this is a good choice for a setting with approx. 100 users where 10 might work concurrently?

Cheers,

Tristan
Sethur
 
Posts: 112
Joined: Thu Jan 16, 2014 11:34 pm

Re: Moving large project to another group

Postby jmoore » Fri Feb 07, 2014 8:06 am

Sethur wrote:On this note: Can you give me a recommendation on the memory settings for the different OMERO.server processes. Currently, I set Blitz to 8 GB and Indexer as well as PixelData to 1 GB. Our total system memory is 16 GB. Do you think this is a good choice for a setting with approx. 100 users where 10 might work concurrently?


Those are certainly reasonable numbers. Going much over 8GB for the Blitz process probably won't help linearly unless you also start tuning other Java options (GC, etc). 1GB for the other 2 processes is also a good starting point. In 5.0.0, there will hopefully be a number of fixes to make the Indexer need less, but if you can spare 1GB for it, it certainly won't hurt.

If you notice that you are having any issues with larger images (3K x 3K or larger), then check the PixelData log. In outside circumstances, you may need to increase the memory for it. Going for 2GB from the start if you can spare might be slightly safer.

Cheers,
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Moving large project to another group

Postby Sethur » Wed Feb 19, 2014 5:25 pm

Hi Josh,

I repeated the move operation of 100 x 55 MB tiffs now after upgrading to 5.0.0-rc2 and it took approx. 4 mins. Also, the server was still responsive during the operation, so I think as long as nobody tries to move extremely large projects, we are fine (assuming this scales linearly).

Cheers,

Tristan
Last edited by Sethur on Mon Feb 24, 2014 5:38 pm, edited 1 time in total.
Sethur
 
Posts: 112
Joined: Thu Jan 16, 2014 11:34 pm

Re: Moving large project to another group

Postby jmoore » Sun Feb 23, 2014 7:49 pm

Thanks for letting us know, Tristan.

Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Previous

Return to User Discussion

Who is online

Users browsing this forum: Google [Bot] and 1 guest