We're Hiring!

How to speed-up Pyramid data creation?

General user discussion about using the OMERO platform to its fullest. Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

How to speed-up Pyramid data creation?

Postby rpoehlmann » Mon Jan 23, 2017 4:42 pm

Dear OME-Team,

unfortunately we are facing once again some issues with Pyramid data creation on our OMERO server.

A user discovered some TIF images that he had imported into OMERO before Christmas and for which no Pyramid data were calculated. When having a closer look at our system it turned out that apparently since quite some time no Pyramid data had been created at all.

After a restart of the OMERO services, Pyramid data creation re-started as well, but now there seems to be an substantial backlog of unprocessed data and overall progress is very slowly.

A (truncated) PixelData Log file is attached.
Code: Select all
JVM Settings:
============
blitz=-Xmx96000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '40', 'strategy': 'percent'})
indexer=-Xmx24000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '10', 'strategy': 'percent'})
pixeldata=-Xmx72000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '30', 'strategy': 'percent'})
repository=-Xmx24000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '10', 'strategy': 'percent'})


PixelData Settings:
Code: Select all
omero.pixeldata.cron=* * * * * ?
omero.pixeldata.repetitions=2
omero.pixeldata.threads=12


Today, I tried to better understand the coresponding parameters and started playiing around with them, however I was not able to substantially improve Pyramid data creation speed.
Our server has 12 CPU cores, 256 GB RAM, is running CentOS 7.1, OMERO 5.2.5-ice35. The OMERO data store resides on a GPFS file-system.
  • I played around with the no. of pixeldata threads, however they seem to stay always the same independent from the actual settings I chose (or I did something wrong which I of course cannot exclude). I can always see 5 _controlling_ threads and a total no. of threads via "ps -eLF" with a name "-Domero.name=PixelData-0" of 42?
  • I understood the "omero.pixeldata.repetitions=2" option in a way that it could be beneficial to deal with a backlog of pixel to process and therefore changed it to "2". However, no effect as far as I can judge compared to the default value before. I tried to find more info about this setting but did not really succeed. What am I doing wrong here?
  • I do not really understand in which order missing pixel data will be processed. When looking at the currently processed data as they appear in the PixelData-0.log file, it seems to be a real mixture of older and newer data, and not e.g. from the oldest towards the current data?
  • Processing seems to be very slow as seen by an 1.9 MB file which took 15 min. to complete (extract from PixelData-0.log):
    Import Date: 2016-12-14 16:21:04
    Dimensions (XY): 1600 x 1200
    Pixels Type: uint8
    Pixels Size (XYZ) (µm):
    84.67 x 84.67
    Z-sections/Timepoints: 1 x 1
    Code: Select all
    du -sh /export/omero/OMERO/ManagedRepository/schibe02_2704/2016-12/14/17-21-03.765/008-HA_HA.tif
    1.9M   /export/omero/OMERO/ManagedRepository/schibe02_2704/2016-12/14/17-21-03.765/008-HA_HA.tif
    [omeronas@omero02 OMERO.server]$ grep "2017-01-23" var/log/PixelData-0.log |  grep "%)" | grep 335562
    2017-01-23 13:25:45,074 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 1/4797 (0%).
    2017-01-23 13:27:13,995 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 480/4797 (9%).
    2017-01-23 13:29:04,254 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 959/4797 (19%).
    2017-01-23 13:30:26,455 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 1438/4797 (29%).
    2017-01-23 13:31:30,678 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 1917/4797 (39%).
    2017-01-23 13:33:22,824 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 2396/4797 (49%).
    2017-01-23 13:34:56,985 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 2875/4797 (59%).
    2017-01-23 13:35:53,342 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 3354/4797 (69%).
    2017-01-23 13:37:37,510 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 3833/4797 (79%).
    2017-01-23 13:39:19,983 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 4312/4797 (89%).
    2017-01-23 13:40:25,867 INFO  [                ome.io.nio.PixelsService] (2-thread-1) Pyramid creation for Pixels:335562 4791/4797 (99%).

    => could it be that we a facing file-system performance issues (although GPFS should be quite fast)?
  • Is there a way to determine the amount of data having no Pyramid data yet? Just curious to know what has accumulated over the past weeks ...
  • Any ideas _why_ the PixelData threads died/stopped working and how this could be prevented/monitored?

Any help/feedback/recommendation would be greatly appreciated to shed some light on this topic.

Many thanks in advance for your support, best,
-Rainer
User avatar
rpoehlmann
 
Posts: 42
Joined: Thu Feb 09, 2012 2:04 pm

Re: How to speed-up Pyramid data creation?

Postby mtbc » Tue Jan 24, 2017 9:53 am

Dear Rainer,

Many good questions. :) I suspect that we will need to draw upon Josh's expertise but perhaps there are also useful hints on https://www.openmicroscopy.org/site/sup ... mance.html.

Could you zip up your server log directory and upload to http://qa.openmicroscopy.org.uk/qa/upload/ and mention when anything especially interesting happened in them? Also, which image formats are you seeing trouble with?

My immediate thoughts with the dying PixelData threads might be shortage of memory or a transient glitch in the underlying mounted storage volume. I wonder what others' experience is.

Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Re: How to speed-up Pyramid data creation?

Postby rpoehlmann » Tue Jan 24, 2017 2:35 pm

Dear Mark,

thanks a lot for your reply.

I uploaded the logs server zip file "OMERO-logs.zip" (see ID 17503, https://www.openmicroscopy.org/qa2/qa/f ... d3d8c62438).

In the meantime Pyramid data creation finished. Nevertheless it would be very valuable to better understand how it can be tweaked and/or prevented from failure.

The affected image formats were TIF files (not extensively tested/validated).

Thanks a lot for your support, cheers,
-Rainer
User avatar
rpoehlmann
 
Posts: 42
Joined: Thu Feb 09, 2012 2:04 pm

Re: How to speed-up Pyramid data creation?

Postby mtbc » Wed Jan 25, 2017 11:09 am

Dear Rainer,

Your server logs certainly show all manner of interesting activity so thank you for those. Regarding pixels service unhappiness the log lines that catch my eye would include those from the morning of 2016-12-13 in Blitz-0.log.4 where we have OutOfMemoryError reports from the server and I/O errors from the database backend. Additionally the PixelData logs often show "Disk quota exceeded" messages. I'd therefore suspect that your server has been suffering from both memory and disk issues at times.

Josh can probably help with your questions about how to speed up the pyramid generation. He might also have some ideas about how to handle any corrupt pyramids you may have been left with, given your "Invalid TIFF file" messages in Blitz-0.log.1. I hope that the above observations help you to figure out why things went wrong in the first place though.

Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Re: How to speed-up Pyramid data creation?

Postby jmoore » Tue Jan 31, 2017 12:36 pm

Hi Rainer,

Trying to catch up:

  • If you think that corrupt Pyramid files have been produced, take a look at `bin/omero admin fixpyramids -h` for a place to get started. (See this thread for an example)
  • Regarding processing ordering, the next image chronologically per user is picked. This is to prevent the queue from being filled by any one user.
  • Before trying to optimize for performance, I'd suggest we take care of the memory issues as well as the disk quotas since it will be difficult to debug otherwise.

All the best,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: How to speed-up Pyramid data creation?

Postby rpoehlmann » Wed Feb 01, 2017 10:23 am

Hi Josh,

thanks for your reply and the info about processing order.

  • I checked (and fixed) corrupted Pyramid files:
    Code: Select all
    [omeronas@omero02 OMERO.server]$ bin/omero admin fixpyramids  /export/omero/OMERO/
    Using session 3b17de97-84c2-40b8-8ed7-a8b99f481db0 (root@localhost:4064). Idle timeout: 200 min. Current group: system
    Removing 197125_pyramid
    Removing 271825_pyramid

    Only a few ones, not really big impact
  • We should not have any disk quota issues
    Code: Select all
    Filesystem Fileset    type             KB      quota      limit   in_doubt    grace
    bc2_fs07   share_omero USR         no limits             

    However, I will doublecheck this with the storage admins!
  • I would be glad to further adjust/improve memory settings in particular if those are short for Pyramid file generation. Any input/recommendation are greatly appreciated.

Best,
-Rainer
User avatar
rpoehlmann
 
Posts: 42
Joined: Thu Feb 09, 2012 2:04 pm

Re: How to speed-up Pyramid data creation?

Postby jmoore » Wed Feb 01, 2017 1:49 pm

Hi Rainer,

what is the current memory settings for your processes:

Code: Select all
$ bin/omero admin jvmcfg
JVM Settings:
============
blitz=-Xmx2576m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
indexer=-Xmx1717m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
pixeldata=-Xmx2576m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
repository=-Xmx1717m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions


~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: How to speed-up Pyramid data creation?

Postby rpoehlmann » Wed Feb 01, 2017 4:30 pm

Hi Josh,

Code: Select all
./bin/omero admin jvmcfg
JVM Settings:
============
blitz=-Xmx96000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '40', 'strategy': 'percent'})
indexer=-Xmx24000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '10', 'strategy': 'percent'})
pixeldata=-Xmx72000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '30', 'strategy': 'percent'})
repository=-Xmx24000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '10', 'strategy': 'percent'})


The server has 256GB RAM in total

Thanks & cheers,
-Rainer
User avatar
rpoehlmann
 
Posts: 42
Joined: Thu Feb 09, 2012 2:04 pm

Re: How to speed-up Pyramid data creation?

Postby jburel » Thu Feb 02, 2017 7:52 pm

Hi Rainer

The server seems to be well provisioned.
The fact that the pyramid generation was getting slow might have been due to the faulty ones
and also due to the backlog of pyramid to generate.
Do you still have pyramids to generate in the queue?
Hopefully the speed will improve when the queue is empty.
Unfortunately we do not have visual tools to monitor that.

Cheers

Jmarie
User avatar
jburel
Team Member
 
Posts: 348
Joined: Thu May 21, 2009 6:38 pm
Location: dundee

Re: How to speed-up Pyramid data creation?

Postby rpoehlmann » Fri Feb 03, 2017 4:32 pm

Hi Jean-Marie,

thanks for your reply. And, it's good to know that JVM settings seems to be more or less fine ;)
The fact that the pyramid generation was getting slow might have been due to the faulty ones
and also due to the backlog of pyramid to generate.

Currently, everything is fine, no Pyramid data backlog.
At least for me, the take home message from our discussion seems to be: if possible try to avoid large backlogs in pyramid data generation. Since our backlog had been caused by a silently dying Pixeldata service, the best would be to detect those cases as soon as possible. And, to regularly scan for faulty ones (cause or effect?).

How to best monitor those scenarios is still puzzling me, but we will find a solution hopefully ...

Anyhow, thanks to everybody for the help, I think we can consider this topic as solved.

Cheers,
-Rainer
User avatar
rpoehlmann
 
Posts: 42
Joined: Thu Feb 09, 2012 2:04 pm


Return to User Discussion

Who is online

Users browsing this forum: Google [Bot] and 1 guest

cron