We're Hiring!

Slow pyramid calculation

Having a problem deploying OMERO? Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

The OMERO.server installation documentation begins here and you can find OMERO.web deployment documentation here.

Slow pyramid calculation

Postby evenhuis » Wed Oct 31, 2018 3:30 am

Hi,

we are experiencing really slow pyramid generation times on some files which has lead to a massive backlog on our server.

I suspect it is due to some files which are either corrupted and/or aren't handled well by bioformats.

Quick questions
  • should this be cross posted to the bioformats forum?
  • is there way to disable pyramid generation for these problems files while we sort it out?

More details below.

Cheers,

Chris

Forum searches
I've had a look through the forums for some tips:

Server setup
From these I ran the 'remove pyramids' script (no problem files) and found an OOM error in the log do I upped the memory for the pixel data. The memory is
Code: Select all
[omero@prdomeapp01 OMERO.server]$ bin/omero admin jvmcfg
JVM Settings:
============
blitz=-Xmx3764m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
indexer=-Xmx2509m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
pixeldata=-Xmx3764m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
repository=-Xmx2509m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOption



It takes about 20hours to create the pyramid, which is surprising because the file is only 16MB.

After the pyramid has been generated there's a bunch if errors which looks like the connection to the database was dropped - perhaps because it took so long the connection timed out?
Code: Select all
2018-10-31 11:51:37,448 WARN  [   o.s.jdbc.support.SQLErrorCodesFactory] (2-thread-4) Error while extracting database product name - falling back to empty error codes
org.springframework.jdbc.support.MetaDataAccessException: Error while extracting DatabaseMetaData; nested exception is org.postgresql.util.PSQLException: This connection has been closed.
snip
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
snip
2018-10-31 11:51:37,485 ERROR [ ome.services.pixeldata.PixelDataHandler] (2-thread-4) Failed to handle pixels 20806


The file itself
The file is a 16MB tif ( 8-bit 11424 x 16640 ) and looks to be generated by Hamamatsu scanner and JPEG copmpressed.

It can easily opened with Preview on MacOS, Inkscape struggles a bit, but Fiji really chokes on it (100% CPU for 10min with no sign of progress). It does eventually open but it must have taken 30min or so.

Demo server
I uploaded the file to the demo server and it seems to being a longtime to create the pyramids there to. The ID is 54506.

The file itself and server logs
The file itself and some the server logs are uploaded to the qa server, the id is 27045
evenhuis
 
Posts: 61
Joined: Tue Jan 30, 2018 4:47 am

Re: Slow pyramid calculation

Postby evenhuis » Wed Oct 31, 2018 10:20 pm

Demo server update
The pyramid was calculated sometime during night for Image ID: 54506. It would interesting to find out how long it took from the PixelData log.

So it seems we are experiencing two problems,
  • Slow generation of the pyramid for this file. It takes imageJ ~30min to open it and I imagine the pyramid calculation involves opening it >10 times.
  • The database connection being dropped on server, maybe due to a timeout someplace
  • The PixelData service halting after the database connection error.

Cheers,

Chris
evenhuis
 
Posts: 61
Joined: Tue Jan 30, 2018 4:47 am

Re: Slow pyramid calculation

Postby Dominik » Fri Nov 02, 2018 2:34 pm

Hi Chris,

thanks for providing the example file. I was able to replicate the issue but I couldn't track it down yet, what the reason for extremly slow pyramid generation (and the high memory consumption) is. We'll working on it and can hopefully provide more information and a bugfix soon.

Regards,
Dominik
User avatar
Dominik
Team Member
 
Posts: 149
Joined: Mon Feb 10, 2014 11:26 am

Re: Slow pyramid calculation

Postby evenhuis » Fri Nov 02, 2018 3:32 pm

Hi Dominick,

Thanks for reply. Is there anyway to skip these files for pyramid generation - I suspect there are about 200 of these in the queue at the moment.

Cheers,

Chris
evenhuis
 
Posts: 61
Joined: Tue Jan 30, 2018 4:47 am

Re: Slow pyramid calculation

Postby jmoore » Mon Nov 05, 2018 8:13 pm

Hi Chris,

joining this thread late: you can skip pending pyramid generations by disabling the pixel data thread, bumping the configuration setting, and then re-enabling:

Code: Select all
$ bin/omero admin ice server disable  PixelData-0
$ bin/omero admin ice server stop  PixelData-0


Code: Select all
postgres=# select * from configuration where name like 'pixel%';
                 name                  | value
---------------------------------------+-------
pixelDataEventLogLoader.v1.current_id | -1
(1 row)

postgres=# select max(id) from pixels;
max
-----
   1
(1 row)

update configuration set value = '1' where name = 'pixelDataEventLogLoader.v1.current_id';
UPDATE 1


Code: Select all
$ bin/omero admin ice server enable  PixelData-0


However, either while you are doing that or even before, you might consider if it's possible to give the pixeldata process more memory. (docs)

Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Slow pyramid calculation

Postby evenhuis » Mon Feb 11, 2019 12:16 pm

Hi Josh,

thanks for the info. We have about a 1,000 of these files so incrementing the counter like that isn't going to be fun.

We have decided to delete these dodgy files, however now we are running into problems with open files. When I select a batch of 32 files to delete this results in 7,000 files being opened and not released for at least an hour. I'll update in the morning if/when they are released. Update - the 7K still have not been released after 8 hour.

What has been happening is that we have been running into the file open limit (~20K for our system) which degrades the system enough to need a restart.

Is this typical for deleting largish amounts of files or is this specific to this dodgy file?

Cheers,

Chris
evenhuis
 
Posts: 61
Joined: Tue Jan 30, 2018 4:47 am

Re: Slow pyramid calculation

Postby mtbc » Tue Feb 12, 2019 9:53 am

Will take this up via viewtopic.php?f=5&t=8680, thanks!

Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland


Return to Installation and Deployment

Who is online

Users browsing this forum: No registered users and 1 guest

cron