We're Hiring!

Delayed availability for .lsm files

General user discussion about using the OMERO platform to its fullest. Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

Delayed availability for .lsm files

Postby VPrevosto » Fri Feb 27, 2015 9:31 pm

Hi everyone,
I have an issue loading/displaying large (~600Mb) mosaic images in .lsm format. After being imported, those images can't be displayed in the client immediately. The thumbnail picture show a clock-like image. However, pictures become available for display 2 to 3 hours later.
I've tested this issue on the workstation running the database, with the same result, so this is not an issue related to CPU, RAM or LAN transfer rate (The database runs on a beefed-up desktop computer with an Intel Core i7-4820K CPU @ 3.70GHz, 4 Cores, 8 Logical Processors, and 64.0 GB installed RAM). Once available, those files are actually displayed very quickly on any remote client.
I went through the Troubleshooting OMERO page, but didn't find a solution.

Here's the diagnostics report
================================================================================
OMERO Diagnostics 5.0.8-ice35-b60
================================================================================

Commands: java -version 1.8.0 (C:\ProgramData\Oracle\Java\javapath\java.EXE -- 2 others)
Commands: python -V 2.7.8 (C:\Anaconda\python.EXE -- 2 others)
Commands: icegridnode --version 3.5.1 (C:\Ice\Ice35\bin\icegridnode.EXE)
Commands: icegridadmin --version 3.5.1 (C:\Ice\Ice35\bin\icegridadmin.EXE)
Commands: psql --version 9.2.10 (C:\Program Files\PostgreSQL\9.2\bin\psql.EXE)

Server: icegridnode running
Server: Blitz-0 active (pid = 2240, enabled)
Server: DropBox inactive (disabled)
Server: FileServer active (pid = 2524, enabled)
Server: Indexer-0 active (pid = 2544, enabled)
Server: MonitorServer inactive (disabled)
Server: OMERO.Glacier2 active (pid = 2684, enabled)
Server: OMERO.IceStorm active (pid = 2712, enabled)
Server: PixelData-0 active (pid = 2756, enabled)
Server: Processor-0 active (pid = 2776, enabled)
Server: Tables-0 active (pid = 2800, enabled)
Server: TestDropBox inactive (enabled)
Server: OMERO.master active (running as LocalSystem)

OMERO: SSL port 4064
OMERO: TCP port 4063

Log dir: D:\OmeroAdmin\var\log exists

Log files: Blitz-0.log 26.0 MB errors=4807 warnings=439
Log files: DropBox.log 4.0 KB errors=15 warnings=0
Log files: FileServer.log 1.0 KB
Log files: Indexer-0.log 11.0 KB
Log files: MonitorServer.log 5.0 KB errors=10 warnings=0
Log files: OMEROweb.log n/a
Log files: PixelData-0.log 15.0 KB
Log files: Processor-0.log 123.0 KB errors=67 warnings=33
Log files: Tables-0.log 131.0 KB errors=67 warnings=33
Log files: TestDropBox.log n/a
Log files: master.err 3.0 KB errors=0 warnings=30
Log files: master.out 0.0 KB
Log files: Total size 26.86 MB


Environment:OMERO_HOME=(unset)
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:OMERO_TEMPDIR=(unset)
Environment:PATH=C:\Anaconda;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\MATLAB\R2014b\runtime\win64;C:\Program Files\MATLAB\R2014b\bin;C:\Program Files\PostgreSQL\9.2\bin;C:\Anaconda\Scripts;C:\Ice\Ice35\bin;C:\Users\Vincent.GeekLU\AppData\Roaming\Dashlane\3.2.3.77451\bin\Firefox_Extension\{442718d9-475e-452a-b3e1-fb1ee16b8e9f}\components;C:\Users\Vincent.GeekLU\AppData\Roaming\Dashlane\3.2.4.78888\bin\Firefox_Extension\{442718d9-475e-452a-b3e1-fb1ee16b8e9f}\components
Environment:PYTHONPATH=C:\Ice\Ice35\python
Environment:ICE_HOME=(unset)
Environment:LD_LIBRARY_PATH=(unset)
Environment:DYLD_LIBRARY_PATH=(unset)

OMERO data dir: 'D:\\OmeroBinaries' Exists? True Is writable? True
OMERO temp dir: 'C:\Users\Vincent.GeekLU\AppData\Roaming\omero\tmp' Exists? True Is writable? True (Size: 0)
OMERO.web status... [NOT STARTED]

Let me know if you have any idea how to fix this.
Thanks!
VPrevosto
 
Posts: 15
Joined: Wed Feb 25, 2015 12:42 am

Re: Delayed availability for .lsm files

Postby jmoore » Mon Mar 02, 2015 11:00 am

VPrevosto wrote:Hi everyone,


Hi Vincent,

I have an issue loading/displaying large (~600Mb) mosaic images in .lsm format. After being imported, those images can't be displayed in the client immediately. The thumbnail picture show a clock-like image. However, pictures become available for display 2 to 3 hours later.


Any image that is over 3000x3000 pixels has an image pyramid generated for it in OMERO. This takes place asynchronously, and you can follow the process in the var/log/PixelData-0.log log file.

I've tested this issue on the workstation running the database, with the same result, so this is not an issue related to CPU, RAM or LAN transfer rate (The database runs on a beefed-up desktop computer with an Intel Core i7-4820K CPU @ 3.70GHz, 4 Cores, 8 Logical Processors, and 64.0 GB installed RAM). Once available, those files are actually displayed very quickly on any remote client. I went through the Troubleshooting OMERO page, but didn't find a solution. ... Let me know if you have any idea how to fix this.


This is most related to https://www.openmicroscopy.org/site/support/omero5.0/sysadmins/troubleshooting.html#outofmemoryerror-permgen-space-errors-in-omero-server-logs. Would you mind sending us the output of:

Code: Select all
bin/omero admin jvmcfg


With as much physical RAM as you have, you'd likely be best served by bumping up the setting for the PixelData service, if not blitz as well. If that doesn't sufficiently lower the times, you could try allowing more pyramids to be generated in parallel via the omero.pixeldata.threads property:

Code: Select all
bin/omero config set omero.pixeldata.threads 4 # or 8


If that still doesn't work, it'd probably be useful to take a look at your PixelData-0.log log file.

All the best,
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Delayed availability for .lsm files

Postby VPrevosto » Tue Mar 03, 2015 2:33 am

Hi Josh,
thanks for your quick and detailed answer. This is very instructive, as my knowledge of Java and JVM processes is 0.

My original JVM settings were:
Code: Select all
blitz=-Xmx7200m -XX:MaxPermSize=1g
indexer=-Xmx4800m -XX:MaxPermSize=1g
pixeldata=-Xmx7200m -XX:MaxPermSize=1g
repository=-Xmx4800m -XX:MaxPermSize=1g


I've tested the image pyramid generation system with more extreme settings (not necessarily what I would use on a permanent basis). These are the settings, post-changes:
Code: Select all
blitz=-Xmx9600m -XX:MaxPermSize=1g # Settings({'max_system_memory': '64000', 'pixeldata': 'manual'})
indexer=-Xmx6400m -XX:MaxPermSize=1g # Settings({'max_system_memory': '64000', 'pixeldata': 'manual'})
pixeldata=-Xmx28800m -XX:MaxPermSize=1g # Settings({'max_system_memory': '64000', 'pixeldata': 'manual', 'heap_size': '28800'})
repository=-Xmx6400m -XX:MaxPermSize=1g # Settings({'max_system_memory': '64000', 'pixeldata': 'manual'})


I've also bumped the pixeldata threads to 8.

All this doesn't seem to help much. I've imported a new image, similar to previous large files, and it took over an hour to process it. I've checked the PixelData-0.log, which shows no error during the process (some errors happen after). Each 10% of pyramid creation takes 7mn for this file, which is comparable with processing time with former settings.
What's more, the dedicated Java process's memory usage is continuously inflating, topping 20Gb in the end, suggesting a few loose ends somewhere. This performance level seems quite poor, considering that the file is only 600Mb. Are my news settings incorrect in some way? Should I test other settings?

I attach the The PixelData-0.log file for your perusal.
PixelData-0.zip
(27.58 KiB) Downloaded 178 times


Thanks for your help!
Best,
Vincent
VPrevosto
 
Posts: 15
Joined: Wed Feb 25, 2015 12:42 am

Re: Delayed availability for .lsm files

Postby jmoore » Tue Mar 03, 2015 7:42 am

Morning, Vincent.

VPrevosto wrote:My original JVM settings were:
Code: Select all
blitz=-Xmx7200m -XX:MaxPermSize=1g
indexer=-Xmx4800m -XX:MaxPermSize=1g
pixeldata=-Xmx7200m -XX:MaxPermSize=1g
repository=-Xmx4800m -XX:MaxPermSize=1g



These actually seem quite reasonable. Bumping blitz and pixeldata by 50-100% would probably be good in general, more only if it's not a burden.

I've tested the image pyramid generation system with more extreme settings (not necessarily what I would use on a permanent basis)....

I've also bumped the pixeldata threads to 8.

All this doesn't seem to help much.


Ok, thanks for trying though. That likely points to an issue with the pyramid generation itself.

I've imported a new image, similar to previous large files, and it took over an hour to process it. I've checked the PixelData-0.log, which shows no error during the process (some errors happen after). Each 10% of pyramid creation takes 7mn for this file, which is comparable with processing time with former settings....

This performance level seems quite poor, considering that the file is only 600Mb. Are my news settings incorrect in some way? Should I test other settings?


Would you be willing to send us one of these files for testing?

What's more, the dedicated Java process's memory usage is continuously inflating, topping 20Gb in the end, suggesting a few loose ends somewhere.


How were you measuring memory usage? In general, the output of top will increase to the limit specified, and never shrink. This is one of the more problematic issues with the JVM, and the reason why you should give each of the processes above exactly as much memory as you can spare to have consumed, since they will be held until the processes are turned off.

Cheers,
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Delayed availability for .lsm files

Postby VPrevosto » Wed Mar 04, 2015 3:35 am

Hi Josh,
many thanks for your prompt replies.
I've shared an example file here:
https://duke.box.com/s/wkxx81yvy5aokhrekbter98xmzx58ngm
(the forum page crashed and reloaded repetitively when I tried to upload it).
About memory usage, I just used Window's Resource Monitor to measure it. Your description of JVM's behavior makes sense and fits what I've seen.
Cheers,
Vincent
VPrevosto
 
Posts: 15
Joined: Wed Feb 25, 2015 12:42 am

Re: Delayed availability for .lsm files

Postby mtbc » Wed Mar 04, 2015 12:52 pm

Vincent,

Thank you for the example file. It does seem to import well on my local system in largely the manner you describe for things at your end, for both OMERO 5.0.8 and the latest working version of 5.1.0. Perhaps it is I/O bound as the server processes indeed did not seem very multi-core, not even running rather faster with settings of,
Code: Select all
omero.jvmcfg.append=-Djj2000.j2k.util.ThreadPool.concurrency=8 -Djj2000.j2k.entropy.encoder.StdEntropyCoder.nthreads=8
omero.pixeldata.threads=8

We'll check if any resident Bio-Formats experts have any ideas on this.

Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Re: Delayed availability for .lsm files

Postby VPrevosto » Wed Mar 04, 2015 4:03 pm

Sounds good. Keep me posted.
Best,
Vincent
VPrevosto
 
Posts: 15
Joined: Wed Feb 25, 2015 12:42 am

Re: Delayed availability for .lsm files

Postby mtbc » Wed Mar 04, 2015 4:12 pm

Vincent,

We do seem to have existing tickets about this matter: http://trac.openmicroscopy.org/ome/ticket/6220 and http://trac.openmicroscopy.org/ome/ticket/6224. But, they are already old, so work may plausibly be further postponed. In the shorter term, as a first step we will look at adding a little more logging to see what is actually going on here: that could point the way forward. I could add you to the cc: on those tickets if you would like to be notified of updates.

Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Re: Delayed availability for .lsm files

Postby mlinkert » Thu Mar 05, 2015 4:15 pm

Hi Vincent,

Thank you again for uploading a file. We are now reviewing a fix for this problem:

https://github.com/openmicroscopy/bioformats/pull/1643

We expect this to be included in the upcoming 5.1.0 release of OMERO. Preliminary tests with this fix show that the image can be viewed within 10 minutes of being imported (instead of multiple hours).

Regards,
-Melissa
User avatar
mlinkert
Team Member
 
Posts: 353
Joined: Fri May 29, 2009 2:12 pm
Location: Southwest Wisconsin

Re: Delayed availability for .lsm files

Postby VPrevosto » Wed Mar 18, 2015 10:03 pm

Hey Melissa and Mark,
that's great news. Thanks for all the hard work on this! (For some reason I missed your forum posts - just read them now).
Best
Vincent
VPrevosto
 
Posts: 15
Joined: Wed Feb 25, 2015 12:42 am


Return to User Discussion

Who is online

Users browsing this forum: Google [Bot] and 1 guest