Delayed availability for .lsm files

[VPrevosto]

Hi everyone,
I have an issue loading/displaying large (~600Mb) mosaic images in .lsm format. After being imported, those images can't be displayed in the client immediately. The thumbnail picture show a clock-like image. However, pictures become available for display 2 to 3 hours later.
I've tested this issue on the workstation running the database, with the same result, so this is not an issue related to CPU, RAM or LAN transfer rate (The database runs on a beefed-up desktop computer with an Intel Core i7-4820K CPU @ 3.70GHz, 4 Cores, 8 Logical Processors, and 64.0 GB installed RAM). Once available, those files are actually displayed very quickly on any remote client.
I went through the Troubleshooting OMERO page, but didn't find a solution.

Here's the diagnostics report
================================================================================
OMERO Diagnostics 5.0.8-ice35-b60
================================================================================

Commands: java -version 1.8.0 (C:\ProgramData\Oracle\Java\javapath\java.EXE -- 2 others)
Commands: python -V 2.7.8 (C:\Anaconda\python.EXE -- 2 others)
Commands: icegridnode --version 3.5.1 (C:\Ice\Ice35\bin\icegridnode.EXE)
Commands: icegridadmin --version 3.5.1 (C:\Ice\Ice35\bin\icegridadmin.EXE)
Commands: psql --version 9.2.10 (C:\Program Files\PostgreSQL\9.2\bin\psql.EXE)

Server: icegridnode running
Server: Blitz-0 active (pid = 2240, enabled)
Server: DropBox inactive (disabled)
Server: FileServer active (pid = 2524, enabled)
Server: Indexer-0 active (pid = 2544, enabled)
Server: MonitorServer inactive (disabled)
Server: OMERO.Glacier2 active (pid = 2684, enabled)
Server: OMERO.IceStorm active (pid = 2712, enabled)
Server: PixelData-0 active (pid = 2756, enabled)
Server: Processor-0 active (pid = 2776, enabled)
Server: Tables-0 active (pid = 2800, enabled)
Server: TestDropBox inactive (enabled)
Server: OMERO.master active (running as LocalSystem)

OMERO: SSL port 4064
OMERO: TCP port 4063

Log dir: D:\OmeroAdmin\var\log exists

Log files: Blitz-0.log 26.0 MB errors=4807 warnings=439
Log files: DropBox.log 4.0 KB errors=15 warnings=0
Log files: FileServer.log 1.0 KB
Log files: Indexer-0.log 11.0 KB
Log files: MonitorServer.log 5.0 KB errors=10 warnings=0
Log files: OMEROweb.log n/a
Log files: PixelData-0.log 15.0 KB
Log files: Processor-0.log 123.0 KB errors=67 warnings=33
Log files: Tables-0.log 131.0 KB errors=67 warnings=33
Log files: TestDropBox.log n/a
Log files: master.err 3.0 KB errors=0 warnings=30
Log files: master.out 0.0 KB
Log files: Total size 26.86 MB


Environment:OMERO_HOME=(unset)
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:OMERO_TEMPDIR=(unset)
Environment:PATH=C:\Anaconda;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\MATLAB\R2014b\runtime\win64;C:\Program Files\MATLAB\R2014b\bin;C:\Program Files\PostgreSQL\9.2\bin;C:\Anaconda\Scripts;C:\Ice\Ice35\bin;C:\Users\Vincent.GeekLU\AppData\Roaming\Dashlane\3.2.3.77451\bin\Firefox_Extension\{442718d9-475e-452a-b3e1-fb1ee16b8e9f}\components;C:\Users\Vincent.GeekLU\AppData\Roaming\Dashlane\3.2.4.78888\bin\Firefox_Extension\{442718d9-475e-452a-b3e1-fb1ee16b8e9f}\components
Environment:PYTHONPATH=C:\Ice\Ice35\python
Environment:ICE_HOME=(unset)
Environment:LD_LIBRARY_PATH=(unset)
Environment:DYLD_LIBRARY_PATH=(unset)

OMERO data dir: 'D:\\OmeroBinaries' Exists? True Is writable? True
OMERO temp dir: 'C:\Users\Vincent.GeekLU\AppData\Roaming\omero\tmp' Exists? True Is writable? True (Size: 0)
OMERO.web status... [NOT STARTED]

Let me know if you have any idea how to fix this.
Thanks!

[jmoore]

Hi everyone,

Hi Vincent,

I have an issue loading/displaying large (~600Mb) mosaic images in .lsm format. After being imported, those images can't be displayed in the client immediately. The thumbnail picture show a clock-like image. However, pictures become available for display 2 to 3 hours later.

Any image that is over 3000x3000 pixels has an image pyramid generated for it in OMERO. This takes place asynchronously, and you can follow the process in the var/log/PixelData-0.log log file.

I've tested this issue on the workstation running the database, with the same result, so this is not an issue related to CPU, RAM or LAN transfer rate (The database runs on a beefed-up desktop computer with an Intel Core i7-4820K CPU @ 3.70GHz, 4 Cores, 8 Logical Processors, and 64.0 GB installed RAM). Once available, those files are actually displayed very quickly on any remote client. I went through the Troubleshooting OMERO page, but didn't find a solution. ... Let me know if you have any idea how to fix this.

This is most related to https://www.openmicroscopy.org/site/support/omero5.0/sysadmins/troubleshooting.html#outofmemoryerror-permgen-space-errors-in-omero-server-logs. Would you mind sending us the output of:

Code: Select all

bin/omero admin jvmcfg


With as much physical RAM as you have, you'd likely be best served by bumping up the setting for the PixelData service, if not blitz as well. If that doesn't sufficiently lower the times, you could try allowing more pyramids to be generated in parallel via the omero.pixeldata.threads property:

Code: Select all

bin/omero config set omero.pixeldata.threads 4 # or 8


If that still doesn't work, it'd probably be useful to take a look at your PixelData-0.log log file.

All the best,
~Josh.

[VPrevosto]

Hi Josh,
thanks for your quick and detailed answer. This is very instructive, as my knowledge of Java and JVM processes is 0.

My original JVM settings were:

Code: Select all

blitz=-Xmx7200m -XX:MaxPermSize=1g
indexer=-Xmx4800m -XX:MaxPermSize=1g
pixeldata=-Xmx7200m -XX:MaxPermSize=1g
repository=-Xmx4800m -XX:MaxPermSize=1g

I've tested the image pyramid generation system with more extreme settings (not necessarily what I would use on a permanent basis). These are the settings, post-changes:

Code: Select all

blitz=-Xmx9600m -XX:MaxPermSize=1g # Settings({'max_system_memory': '64000', 'pixeldata': 'manual'})
indexer=-Xmx6400m -XX:MaxPermSize=1g # Settings({'max_system_memory': '64000', 'pixeldata': 'manual'})
pixeldata=-Xmx28800m -XX:MaxPermSize=1g # Settings({'max_system_memory': '64000', 'pixeldata': 'manual', 'heap_size': '28800'})
repository=-Xmx6400m -XX:MaxPermSize=1g # Settings({'max_system_memory': '64000', 'pixeldata': 'manual'})

I've also bumped the pixeldata threads to 8.

All this doesn't seem to help much. I've imported a new image, similar to previous large files, and it took over an hour to process it. I've checked the PixelData-0.log, which shows no error during the process (some errors happen after). Each 10% of pyramid creation takes 7mn for this file, which is comparable with processing time with former settings.
What's more, the dedicated Java process's memory usage is continuously inflating, topping 20Gb in the end, suggesting a few loose ends somewhere. This performance level seems quite poor, considering that the file is only 600Mb. Are my news settings incorrect in some way? Should I test other settings?

I attach the The PixelData-0.log file for your perusal.

PixelData-0.zip

(27.58 KiB) Downloaded 177 times

Thanks for your help!
Best,
Vincent

[jmoore]

Morning, Vincent.

My original JVM settings were:

Code: Select all

blitz=-Xmx7200m -XX:MaxPermSize=1g
indexer=-Xmx4800m -XX:MaxPermSize=1g
pixeldata=-Xmx7200m -XX:MaxPermSize=1g
repository=-Xmx4800m -XX:MaxPermSize=1g

These actually seem quite reasonable. Bumping blitz and pixeldata by 50-100% would probably be good in general, more only if it's not a burden.

I've tested the image pyramid generation system with more extreme settings (not necessarily what I would use on a permanent basis)....

I've also bumped the pixeldata threads to 8.

All this doesn't seem to help much.

Ok, thanks for trying though. That likely points to an issue with the pyramid generation itself.

I've imported a new image, similar to previous large files, and it took over an hour to process it. I've checked the PixelData-0.log, which shows no error during the process (some errors happen after). Each 10% of pyramid creation takes 7mn for this file, which is comparable with processing time with former settings....

This performance level seems quite poor, considering that the file is only 600Mb. Are my news settings incorrect in some way? Should I test other settings?

Would you be willing to send us one of these files for testing?

What's more, the dedicated Java process's memory usage is continuously inflating, topping 20Gb in the end, suggesting a few loose ends somewhere.

How were you measuring memory usage? In general, the output of top will increase to the limit specified, and never shrink. This is one of the more problematic issues with the JVM, and the reason why you should give each of the processes above exactly as much memory as you can spare to have consumed, since they will be held until the processes are turned off.

Cheers,
~Josh.

[VPrevosto]

Hi Josh,
many thanks for your prompt replies.
I've shared an example file here:
https://duke.box.com/s/wkxx81yvy5aokhrekbter98xmzx58ngm
(the forum page crashed and reloaded repetitively when I tried to upload it).
About memory usage, I just used Window's Resource Monitor to measure it. Your description of JVM's behavior makes sense and fits what I've seen.
Cheers,
Vincent

[mtbc]

Vincent,

Thank you for the example file. It does seem to import well on my local system in largely the manner you describe for things at your end, for both OMERO 5.0.8 and the latest working version of 5.1.0. Perhaps it is I/O bound as the server processes indeed did not seem very multi-core, not even running rather faster with settings of,

Code: Select all

omero.jvmcfg.append=-Djj2000.j2k.util.ThreadPool.concurrency=8 -Djj2000.j2k.entropy.encoder.StdEntropyCoder.nthreads=8
omero.pixeldata.threads=8


We'll check if any resident Bio-Formats experts have any ideas on this.

Cheers,
Mark

[VPrevosto]

Sounds good. Keep me posted.
Best,
Vincent

[mtbc]

Vincent,

We do seem to have existing tickets about this matter: http://trac.openmicroscopy.org/ome/ticket/6220 and http://trac.openmicroscopy.org/ome/ticket/6224. But, they are already old, so work may plausibly be further postponed. In the shorter term, as a first step we will look at adding a little more logging to see what is actually going on here: that could point the way forward. I could add you to the cc: on those tickets if you would like to be notified of updates.

Cheers,
Mark

[mlinkert]

Hi Vincent,

Thank you again for uploading a file. We are now reviewing a fix for this problem:

https://github.com/openmicroscopy/bioformats/pull/1643

We expect this to be included in the upcoming 5.1.0 release of OMERO. Preliminary tests with this fix show that the image can be viewed within 10 minutes of being imported (instead of multiple hours).

Regards,
-Melissa

[VPrevosto]

Hey Melissa and Mark,
that's great news. Thanks for all the hard work on this! (For some reason I missed your forum posts - just read them now).
Best
Vincent