Blitz Log Errors

[Dominik]

Thanks for providing some example images. I'll try to reproduce the errors with your files today.
In order to put all the information together I also created a ticket for this issue:
https://trac.openmicroscopy.org/ome/ticket/12687

Cheers,
Dominik

[Dominik]

Unfortunately I still wasn't able to reproduce these errors. At the moment I can only suggest to update to the latest 5.0.6 version and check if the errors still appear with that version. According to the log file it looks like Insight tries to close a session which already has been closed before, so I think it's not a critical error.

Cheers,
Dominik

[brossetti]

hmmm, the plot thickens. I will perform the upgrade and do a bit more testing. Thanks for looking into this!

-Blair

[Dominik]

Yes, please let us know, if you continue seeing these errors. I keep my eyes open, if I stumble upon them elsewhere.

Cheers,
Dominik

[Dominik]

Hi Blair,

any update on this issue from your side? If not, I'll close the ticket, as I still can't reproduce the warning messages with the images you provided.

Cheers,
Dominik

[brossetti]

Hi Dominik,

I had a chance to update to 5.0.6 the last week. Unfortunately, we are still seeing the same errors:

Code: Select all

2015-01-30 16:53:21,569 ERROR [o.s.blitz.impl.RawFileStoreI$OpsDelegate] (2-thread-2) Error on close, stage1
2015-01-30 16:53:21,573 ERROR [                      omero.cmd.SessionI] (2-thread-2) Error destroying servant: f00f567a-e69e-4c14-ac74-12aa01217c40omero.api.RawFileStore=ome.services.blitz.impl.RawFileStoreI$OpsDelegate@486847e2

I am trying to track down what was happening during the times reported. As far as I can tell, there was no upload activity from users on 1/30/2015.

Code: Select all

grep ERROR Blitz-0.log  | grep stage1 | cut -f1-2 -d: | uniq -c
     36 2015-01-30 16:53

Since the errors occur during the workday, I am assuming the they are coming from user interactions. However, it does not appear that the problem is related to uploads.

The mystery continues...

Please let me know if you have any tests that might help narrow down the possible issues.

Thanks,
Blair

[manics]

Hi Blair

Could you upload your entire server logs directory? Zip it up and upload to:
https://www.openmicroscopy.org/qa2/qa/upload/

It would also be helpful if you could give us some of the timestamps when you noticed errors occuring at the same time as uploads, and when you know uploads weren't in progress.

Could you also tell us a bit more about your server configuration e.g. OS including version, hardware setup such as network mounts, etc.

Thanks, SImon

[brossetti]

Hi Simon,

I just uploaded the log files. Here is the latest diagnostic report:

Code: Select all

================================================================================
OMERO Diagnostics 5.0.6-ice35-b53
================================================================================

WARNING:omero.util.UpgradeCheck:UPGRADE AVAILABLE:Please upgrade to 5.0.7 See http://trac.openmicroscopy.org.uk/omero for the latest version

Commands:   java -version                  1.7.0     (/usr/bin/java)
Commands:   python -V                      2.6.6     (/usr/bin/python)
Commands:   icegridnode --version          3.5.1     (/usr/bin/icegridnode)
Commands:   icegridadmin --version         3.5.1     (/usr/bin/icegridadmin)
Commands:   psql --version                 9.3.5     (/usr/bin/psql)

Server:     icegridnode                    running
Server:     Blitz-0                        active (pid = 15354, enabled)
Server:     DropBox                        active (pid = 15384, enabled)
Server:     FileServer                     active (pid = 15377, enabled)
Server:     Indexer-0                      active (pid = 15372, enabled)
Server:     MonitorServer                  active (pid = 15393, enabled)
Server:     OMERO.Glacier2                 active (pid = 15405, enabled)
Server:     OMERO.IceStorm                 active (pid = 15394, enabled)
Server:     PixelData-0                    active (pid = 15395, enabled)
Server:     Processor-0                    active (pid = 15414, enabled)
Server:     Tables-0                       active (pid = 15397, enabled)
Server:     TestDropBox                    inactive (enabled)

OMERO:      SSL port                       4064
OMERO:      TCP port                       4063

Log dir:    /home/omero/OMERO.server/var/log exists

Log files:  Blitz-0.log                    32.0 MB       errors=72   warnings=267
Log files:  DropBox.log                    1.0 KB
Log files:  FileServer.log                 0.0 KB
Log files:  Indexer-0.log                  204.0 KB
Log files:  MonitorServer.log              1.0 KB
Log files:  OMEROweb.log                   37.0 KB       errors=0    warnings=4
Log files:  OMEROweb_request.log           0.0 KB        errors=0    warnings=4
Log files:  PixelData-0.log                100.0 KB
Log files:  Processor-0.log                0.0 KB
Log files:  Tables-0.log                   0.0 KB
Log files:  TestDropBox.log                n/a
Log files:  master.err                     0.0 KB
Log files:  master.out                     0.0 KB
Log files:  Total size                     32.76 MB


Environment:OMERO_HOME=(unset)
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:OMERO_TEMPDIR=(unset)
Environment:PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
Environment:PYTHONPATH=(unset)
Environment:ICE_HOME=(unset)
Environment:LD_LIBRARY_PATH=(unset)
Environment:DYLD_LIBRARY_PATH=(unset)

OMERO data dir: '/OMERO'   Exists? True   Is writable? True
OMERO temp dir: '/home/omero/omero/tmp'   Exists? True   Is writable? True   (Size: 0)

Here is a little background on our setup:

We have a CentOS 6.5 virtual machine with an NFS mounted 48TB high-availability Synology cluster. Based on recommendations on this thread, I symlinked the /OMERO directory towards the NFS mount. The connection to the NFS mount is monitored by a cron job. The VMware ESX server is connected to the Synology cluster through a 10 gigabit switch followed by a 1 gigabit switch.

I will continue to look into events corresponding to error timestamps.

Thanks!
Blair

[Dominik]

Thanks for uploading your new 5.0.6 log files. I was hoping the warnings will disappear after an update, but unfortunately this wasn't the case. I still don't have an idea about the cause, yet; but we'll continue to investigate.

Cheers,
Dominik

[PaulVanSchayck]

Hi all,

I'll join in on the discussion as I'm seeing similar errors in my log. Especially the "Error on close, stage1" error. I've pasted a set here:
http://pastebin.com/XYk80z1Y

I've not been able to link them to specific behaviour or events.

What's similar in our setup compared to Blair's is the fact that we use NFS to store our binary repository. Could it be this is an issue related to file locking? I'm also not entirely sure we've seen this issue from the first 5.0 release, it may have been introduced between 5.0 and 5.0.3 (the version we are running).

Thanks,

Paul