Page 1 of 4

Problem after upgrade 5.06 to 5.1

PostPosted: Mon Apr 13, 2015 12:07 pm
by phm
Hi,

After upgrading my server to 5.1 I can't start omero.
omero admin diagnostics :
================================================================================
OMERO Diagnostics 5.1.0-ice35-b40
================================================================================

Commands: java -version 1.7.0 (/usr/bin/java)
Commands: python -V 2.6.6 (/usr/bin/python)
Commands: icegridnode --version 3.5.1 (/usr/bin/icegridnode)
Commands: icegridadmin --version 3.5.1 (/usr/bin/icegridadmin)
Commands: psql --version 9.3.6 (/usr/pgsql-9.3/bin/psql -- 2 others)

Server: icegridnode running
Server: Blitz-0 active (pid = 9795, enabled)
Server: DropBox active (pid = 9815, enabled)
Server: FileServer active (pid = 9823, enabled)
Server: Indexer-0 active (pid = 9824, enabled)
Server: MonitorServer active (pid = 9825, enabled)
Server: OMERO.Glacier2 active (pid = 9827, enabled)
Server: OMERO.IceStorm active (pid = 9838, enabled)
Server: PixelData-0 active (pid = 9839, enabled)
Server: Processor-0 active (pid = 9845, enabled)
Server: Tables-0 inactive (disabled)
Server: TestDropBox inactive (enabled)

Log dir: /opt/OMERO.server/var/log exists

Log files: Blitz-0.log 158.0 KB errors=32 warnings=5
Log files: DropBox.log 9.0 KB errors=8 warnings=5
Log files: FileServer.log 1.0 KB
Log files: Indexer-0.log 8.0 MB errors=2196 warnings=549
Log files: MonitorServer.log 3.0 KB
Log files: OMEROweb.log 0.0 KB
Log files: OMEROweb_request.log 0.0 KB
Log files: PixelData-0.log 4.0 MB errors=1096 warnings=274
Log files: Processor-0.log 65.0 KB errors=34 warnings=18
Log files: Tables-0.log n/a
Log files: TestDropBox.log n/a
Log files: master.err 4.0 KB errors=6 warnings=8
Log files: master.out 0.0 KB
Log files: Total size 13.66 MB


Environment:OMERO_HOME=(unset)
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:OMERO_TEMPDIR=/tmp
Environment:PATH=/usr/pgsql-9.3/bin/:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/dell/srvadmin/bin:/opt/OMERO.server/bin:/opt/dell/srvadmin/sbin:/usr/java/latest/bin
Environment:PYTHONPATH=(unset)
Environment:ICE_HOME=(unset)
Environment:LD_LIBRARY_PATH=(unset)
Environment:DYLD_LIBRARY_PATH=(unset)

OMERO SSL port:4064
OMERO TCP port:4063
OMERO data dir:'/OMERO' Exists? True Is writable? True
OMERO temp dir:'/tmp/omero/tmp' Exists? True Is writable? True (Size: 0)

JVM settings: Blitz -Xmx6400m -XX:MaxPermSize=1g # Settings({'system_memory': '32000', 'percent': '20'})
JVM settings: Indexer -Xmx3200m -XX:MaxPermSize=1g # Settings({'system_memory': '32000'})
JVM settings: Pixeldata -Xmx4800m -XX:MaxPermSize=1g # Settings({'system_memory': '32000'})
JVM settings: Repository -Xmx3200m -XX:MaxPermSize=1g # Settings({'system_memory': '32000'})

OMERO.web status... [RUNNING] (PID 9601)



And master.err
2015-04-13 14:00:35,863 INFO [ ome.services.util.ServiceHandler] (l.Server-6) Executor.doWork -- ome.services.sessions.SessionManagerImpl.executeCheckPasswordRO[root]
2015-04-13 14:00:35,863 INFO [ ome.services.util.ServiceHandler] (l.Server-6) Args: [null, InternalSF@702701196]
2015-04-13 14:00:35,867 INFO [ ome.security.basic.EventHandler] (l.Server-6) Auth: user=0,group=0,event=null(Sessions),sess=14417ecb-d786-40b6-a078-6f5dc159852f
2015-04-13 14:00:35,883 INFO [ org.perf4j.TimingLogger] (l.Server-6) start[1428926435863] time[19] tag[omero.call.success.ome.services.sessions.SessionManagerImpl$8.doWork]
2015-04-13 14:00:35,883 INFO [ ome.services.util.ServiceHandler] (l.Server-6) Rslt: false
2015-04-13 14:00:35,885 INFO [ ome.services.util.ServiceHandler] (l.Server-6) Executor.doWork -- java.lang.String.root[]
2015-04-13 14:00:35,885 INFO [ ome.services.util.ServiceHandler] (l.Server-6) Args: [null, InternalSF@702701196]
2015-04-13 14:00:35,889 INFO [ ome.security.basic.EventHandler] (l.Server-6) Auth: user=0,group=0,event=null(Internal),sess=14417ecb-d786-40b6-a078-6f5dc159852f
2015-04-13 14:00:35,898 INFO [ org.perf4j.TimingLogger] (l.Server-6) start[1428926435885] time[13] tag[omero.call.success.ome.services.blitz.fire.PermissionsVerifierI$1.doWork]
2015-04-13 14:00:35,899 INFO [ ome.services.util.ServiceHandler] (l.Server-6) Rslt: null
[omero@omero OMERO.server]$ tail var/log/master.err
from tables.utilsExtension import getPyTablesVersion, getHDF5Version
ImportError: libhdf5.so.6: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File "lib/python/runTables.py", line 15, in <module>
import omero.tables
File "/opt/OMERO.server-5.1.0-ice35-b40/lib/python/omero/tables.py", line 32, in <module>
tables = __import__("tables") # Pytables
File "/usr/lib64/python2.6/site-packages/tables/__init__.py", line 30, in <module>
from tables.utilsExtension import getPyTablesVersion, getHDF5Version
ImportError: libhdf5.so.6: cannot open shared object file: No such file or directory


Any idea ??
Philippe

Re: Problem after upgrade 5.06 to 5.1

PostPosted: Mon Apr 13, 2015 12:42 pm
by jmoore
Could you send us more or all of var/log/Blitz-0.log?

Thanks,
~Josh.

Re: Problem after upgrade 5.06 to 5.1

PostPosted: Mon Apr 13, 2015 12:58 pm
by phm
jmoore wrote:Could you send us more or all of var/log/Blitz-0.log?

Thanks,
~Josh.


tail var/log/Blitz-0.log
2015-04-13 14:30:06,641 INFO [ ome.services.util.ServerVersionCheck] ( main) -------------------------------------------------
2015-04-13 14:30:06,641 INFO [ ome.services.util.ServerVersionCheck] ( main) Stopping OMERO...
2015-04-13 14:30:06,641 INFO [ ome.services.util.ServerVersionCheck] ( main) -------------------------------------------------
2015-04-13 14:30:06,641 INFO [ ome.services.pixeldata.PixelDataThread] ( main) Shutting down PixelDataThread
2015-04-13 14:30:06,641 INFO [ ome.services.eventlogs.EventLogQueue] ( main) Shutting down EventLogLoader
2015-04-13 14:30:06,641 INFO [ ome.services.fulltext.FullTextThread] ( main) Shutting down Full-Text Indexer
2015-04-13 14:30:06,642 INFO [ ome.services.eventlogs.EventLogQueue] ( main) Shutting down EventLogLoader
2015-04-13 14:30:06,642 INFO [ o.s.scheduler.SchedulerFactoryBean] ( main) Shutting down Quartz Scheduler
2015-04-13 14:30:06,645 INFO [.s.ShutdownSafeEhcacheManagerFactoryBean] ( main) Shutting down EHCache CacheManager
2015-04-13 14:30:06,646 INFO [ ome.services.blitz.Entry] ( main) Finished shutdown.

Re: Problem after upgrade 5.06 to 5.1

PostPosted: Mon Apr 13, 2015 1:39 pm
by sbesson
Hi Philippe,

We may need lines before the server shutdown to find the nature of the error. Would it be possible for you to zip the Blitz logs and upload them to https://www.openmicroscopy.org/qa2/qa/upload/.

Best
Sebastien

Re: Problem after upgrade 5.06 to 5.1

PostPosted: Mon Apr 13, 2015 1:49 pm
by phm
sbesson wrote:Hi Philippe,

We may need lines before the server shutdown to find the nature of the error. Would it be possible for you to zip the Blitz logs and upload them to https://www.openmicroscopy.org/qa2/qa/upload/.

Best
Sebastien


Hi Sebastien,

I uploaded the Blitz log

Thanks

Philippe

Re: Problem after upgrade 5.06 to 5.1

PostPosted: Mon Apr 13, 2015 2:00 pm
by jmoore
Hi Philippe,

thanks for the log. It looks like there were some initial issues with the database having not been upgraded, but from what I can tell, the server is up and running:
Code: Select all
10994$ grep -E "(ERROR|Ready)" Blitz-0.log
2015-04-13 12:50:19,233 ERROR [          ome.services.util.DBPatchCheck] (      main)
2015-04-13 12:50:19,240 ERROR [                ome.services.blitz.Entry] (      main) Error on startup.
2015-04-13 13:04:12,893 ERROR [          ome.services.util.DBPatchCheck] (      main)
2015-04-13 13:04:12,899 ERROR [                ome.services.blitz.Entry] (      main) Error on startup.
2015-04-13 13:27:35,802 ERROR [          ome.services.util.DBPatchCheck] (      main)
2015-04-13 13:27:35,815 ERROR [                ome.services.blitz.Entry] (      main) Error on startup.
2015-04-13 13:37:04,513 ERROR [          ome.services.util.DBPatchCheck] (      main)
2015-04-13 13:37:04,519 ERROR [                ome.services.blitz.Entry] (      main) Error on startup.
2015-04-13 13:49:34,099 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.1.0-ice35-b40 Ready.
2015-04-13 14:00:09,295 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.1.0-ice35-b40 Ready.
2015-04-13 14:19:35,436 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.1.0-ice35-b40 Ready.
2015-04-13 14:29:28,805 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.1.0-ice35-b40 Ready.
2015-04-13 14:30:06,552 ERROR [                      omero.cmd.SessionI] (l.Server-3) Error destroying servant: 416cac6c-13eb-411f-a61f-ac77e85bf3ebomero.grid.SharedResources=ome.services.blitz.impl.SharedResourcesI@4190a6ce
2015-04-13 15:01:16,722 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 5.1.0-ice35-b40 Ready.


What isn't working for you?
~Josh.

Re: Problem after upgrade 5.06 to 5.1

PostPosted: Mon Apr 13, 2015 2:12 pm
by phm
jmoore wrote:Hi Philippe,

thanks for the log. It looks like there were some initial issues with the database having not been upgraded, but from what I can tell, the server is up and running:

What isn't working for you?
~Josh.


Hi Josh,

I did the database upgrade and cleaning and now the omero server is ok. However, the web part is down ???
Should I refactor the apache file ???


Thanks

Philippe

Re: Problem after upgrade 5.06 to 5.1

PostPosted: Mon Apr 13, 2015 6:20 pm
by sbesson
Hi Philippe,

can you tell us more about your OMERO.web setup? Was the stanza regenerated using `bin/omero web config` as part of the upgrade? Is there anything either in the Web logs under var/log in the server or under the nginx/apache logs?

Best,
Sebastien

Re: Problem after upgrade 5.06 to 5.1

PostPosted: Tue Apr 14, 2015 8:13 am
by phm
sbesson wrote:Hi Philippe,

can you tell us more about your OMERO.web setup? Was the stanza regenerated using `bin/omero web config` as part of the upgrade? Is there anything either in the Web logs under var/log in the server or under the nginx/apache logs?

Best,
Sebastien

Hi Sebastien,

I think that's a apache config problem more than omero web config. My stanza was regenerated by omero web config apache > web_omero.conf (to be loaded after ssl.conf)
Code: Select all
###
# apache config for omero
# this file should be loaded *after* ssl.conf
#
# -D options to control configurations
#  OmeroWebClientRedirect - redirect / to /omero/webclient
#  OmeroWebAdminRedirect  - redirect / to /omero/webadmin
#  OmeroForceSSL - redirect all http requests to https

###
### Example SSL stanza for OMERO.web created 2015-04-14 09:50:59.536216
###

# Eliminate overlap warnings with the default ssl vhost
# Requires SNI (http://wiki.apache.org/httpd/NameBasedSSLVHostsWithSNI)
# support most later versions of mod_ssl and OSes will support it if you see
# "You should not use name-based virtual hosts in conjunction with SSL!!"
# or similar start apache with -D DISABLE_SNI and modify ssl.conf
#<IfDefine !DISABLE_SNI>
#  NameVirtualHost *:443
#</IfDefine>
#
## force https/ssl
#<IfDefine OmeroForceSSL>
#  RewriteEngine on
#  RewriteCond %{HTTPS} !on
#  RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [L]
#</IfDefine>
#
#<VirtualHost _default_:443>
#
#  ErrorLog logs/ssl_error_log
#  TransferLog logs/ssl_access_log
#  LogLevel warn
#
#  SSLEngine on
#  SSLProtocol all -SSLv2
#  SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
#  SSLCertificateFile /etc/pki/tls/certs/server.crt
#  SSLCertificateKeyFile /etc/pki/tls/private/server.key
#
#  # SSL Protocol Adjustments:
#  SetEnvIf User-Agent ".*MSIE.*" \
#    nokeepalive ssl-unclean-shutdown \
#    downgrade-1.0 force-response-1.0
#
#  # Per-Server Logging:
#  CustomLog logs/ssl_request_log \
#    "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
#
#</VirtualHost>

###
### Stanza for OMERO.web created 2015-04-14 09:50:59.536216
###

FastCGIExternalServer "/opt/OMERO.server/var/omero.fcgi" -host 127.0.0.1:4080 -idle-timeout 60

<Directory "/opt/OMERO.server/var">
    Options -Indexes FollowSymLinks
    Order allow,deny
    Allow from all
</Directory>

<Directory "/opt/OMERO.server/lib/python/omeroweb/static">
    Options -Indexes FollowSymLinks
    Order allow,deny
    Allow from all
</Directory>

<Directory "/opt/OMERO.server/etc/templates/error">
    Options -Indexes FollowSymLinks
    Order allow,deny
    Allow from all
</Directory>

<Location /omero/>
    ErrorDocument 500 /omero/error/maintainance.html
</Location>

Alias /omero/error /opt/OMERO.server/etc/templates/error
Alias /static /opt/OMERO.server/lib/python/omeroweb/static
Alias /omero "/opt/OMERO.server/var/omero.fcgi/"


The OMEROweb.log is empty
When I start omero web I get these messages :
Code: Select all
Post-processed 'omeroweb.viewer.min.css' as 'omeroweb.viewer.min.css'
Post-processed 'omeroweb.viewer.min.js' as 'omeroweb.viewer.min.js'

0 static files copied, 745 unmodified, 2 post-processed.
Starting OMERO.web... [OK]


And apache error_log
Code: Select all
[Tue Apr 14 10:06:13 2015] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:httpd_t:s0
[Tue Apr 14 10:06:13 2015] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Apr 14 10:06:13 2015] [error] (13)Permission denied: FastCGI: apr_dir_open() failed
[Tue Apr 14 10:06:13 2015] [notice] Digest: generating secret for digest authentication ...
[Tue Apr 14 10:06:13 2015] [notice] Digest: done
[Tue Apr 14 10:06:13 2015] [error] (13)Permission denied: FastCGI: apr_dir_open() failed
[Tue Apr 14 10:06:13 2015] [notice] FastCGI: process manager initialized (pid 3548)
[Tue Apr 14 10:06:13 2015] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Tue Apr 14 10:06:13 2015] [notice] mod_python: using mutex_directory /tmp
[Tue Apr 14 10:06:13 2015] [notice] Apache/2.2.15 (Unix) DAV/2 mod_fastcgi/2.4.6 PHP/5.3.3 mod_python/3.3.1 Python/2.6.6 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations


It seems that I have a problem with FastCGI authorization ???
And in the apache ssl_error_log
Code: Select all
[Tue Apr 14 10:13:42 2015] [error] [client 192.168.188.18] (13)Permission denied: access to /omero denied



Best

Philippe

Re: Problem after upgrade 5.06 to 5.1

PostPosted: Tue Apr 14, 2015 9:19 am
by sbesson
Hi Philippe

the following line in the error log:

Code: Select all
[Tue Apr 14 10:13:42 2015] [error] [client 192.168.188.18] (13)Permission denied: access to /omero denied


hints towards a file permissions issue. Could you check the Apache agent can access the /opt/OMERO.server/var directory? If not, you may want to:

  • either fix the permissions of this folder to allow Apache to read it.
  • or move omero.fcgi to another location that Apache can access, e.g. `/var/run/omero.fcgi`.

Best regards
Sebastien