Upgrade 5.2.1 to 5.2.2 - OMEROweb fails to start

[dsudar]

Hi all,
I had been looking forward to 5.2.2 since it has many improvements for my use. Most of the upgrade from 5.2.1 on Ubuntu 14.04.4 went without a problem and OMERO itself is running fine. However when starting OMERO.web, it claims that it started but diagnostics says otherwise:

omero@quid:~$ omero web start
Post-processed 'omeroweb.viewer.min.css' as 'omeroweb.viewer.min.css'
Post-processed 'omeroweb.viewer.min.js' as 'omeroweb.viewer.min.js'

0 static files copied to '/home/omero/OMERO.server/lib/python/omeroweb/static', 764 unmodified, 2 post-processed.
Clearing expired sessions. This may take some time... [OK]
Starting OMERO.web... [OK]
omero@quid:~$ omero admin diagnostics

================================================================================
OMERO Diagnostics 5.2.2-ice35-b17
================================================================================

Commands: java -version 1.7.0 (/usr/bin/java)
Commands: python -V 2.7.6 (/usr/bin/python)
Commands: icegridnode --version 3.5.1 (/usr/bin/icegridnode)
Commands: icegridadmin --version 3.5.1 (/usr/bin/icegridadmin)
Commands: psql --version 9.4.5 (/usr/bin/psql -- 2 others)

Server: icegridnode running
Server: Blitz-0 active (pid = 20957, enabled)
Server: DropBox active (pid = 20980, enabled)
Server: FileServer active (pid = 20987, enabled)
Server: Indexer-0 active (pid = 20989, enabled)
Server: MonitorServer active (pid = 20990, enabled)
Server: OMERO.Glacier2 active (pid = 20988, enabled)
Server: OMERO.IceStorm active (pid = 21012, enabled)
Server: PixelData-0 active (pid = 20993, enabled)
Server: Processor-0 active (pid = 21014, enabled)
Server: Tables-0 active (pid = 21030, enabled)
Server: TestDropBox inactive (enabled)

Log dir: /home/omero/OMERO.server/var/log exists
Log files: Blitz-0.log 1.0 MB errors=18 warnings=1
Log files: DropBox.log 1.0 KB
Log files: FileServer.log 0.0 KB
Log files: Indexer-0.log 1.0 KB
Log files: MonitorServer.log 1.0 KB
Log files: OMEROweb.lock 0.0 KB
Log files: OMEROweb.log 1.0 KB errors=0 warnings=6
Log files: OMEROweb_brokenrequest.lock 0.0 KB
Log files: OMEROweb_brokenrequest.log 0.0 KB
Log files: PixelData-0.log 3.0 KB
Log files: Processor-0.log 63.0 KB errors=18 warnings=0
Log files: Tables-0.log 0.0 KB
Log files: TestDropBox.log n/a
Log files: master.err 0.0 KB
Log files: master.out 0.0 KB
Log files: Total size 1.30 MB


Environment:OMERO_HOME=/home/omero/OMERO.server
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:OMERO_USERDIR=(unset)
Environment:OMERO_TMPDIR=(unset)
Environment:PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/java-7-openjdk-amd64/bin:/usr/share/Ice-3.5.1:/usr/lib/postgresql/9.4/bin:/home/omero/OMERO.server/bin:/home/omero/omero-csv-tools/bin
Environment:PYTHONPATH=/usr/lib/pymodules/python2.7:
Environment:ICE_HOME=/usr/share/Ice-3.5.1
Environment:LD_LIBRARY_PATH=/usr/share/java:/usr/lib:
Environment:DYLD_LIBRARY_PATH=(unset)

OMERO SSL port:4064
OMERO TCP port:4063
OMERO data dir:'/data/OMERO' Exists? True Is writable? True
OMERO temp dir:'/home/omero/omero/tmp' Exists? True Is writable? True (Size: 0)

JVM settings: Blitz-${index} -Xmx7200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Indexer-${index} -Xmx4800m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: PixelData-${index} -Xmx7200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Repository-${index} -Xmx4800m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions

OMERO.web status... [NOT STARTED]
Django version: 1.8.8
omero@quid:~$ omero web status
OMERO.web status... [NOT STARTED]
omero@quid:~$

In nginx's error.log it says:

2016/03/01 18:07:28 [error] 22000#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 99.
47.70.237, server: quid, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:4080/", host: "quid.lbl.gov"
2016/03/01 18:07:33 [error] 22000#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 99.
47.70.237, server: quid, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:4080/", host: "quid.lbl.gov"
2016/03/01 18:14:41 [error] 23927#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 99.
47.70.237, server: quid, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:4080/", host: "quid.lbl.gov"
2016/03/01 18:20:13 [error] 23927#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 99.
47.70.237, server: quid, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:4080/", host: "quid.lbl.gov"

and OMEROweb.log only has some warnings:

2016-03-02 01:51:30,705 WARNI [ py.warnings] (proc.21270) <module>():9 /home/omero/OMERO.server/
lib/python/pipeline/utils.py:11: RemovedInDjango19Warning: django.utils.importlib will be removed in Django 1.9.
from django.utils import importlib

2016-03-02 01:51:31,510 WARNI [ django.security.SuspiciousSession] (proc.21276) decode():108 Session data corrupted
2016-03-02 01:51:31,510 WARNI [ django.security.SuspiciousSession] (proc.21276) decode():108 Session data corrupted
2016-03-02 01:52:25,553 WARNI [ py.warnings] (proc.21717) <module>():9 /home/omero/OMERO.server/
lib/python/pipeline/utils.py:11: RemovedInDjango19Warning: django.utils.importlib will be removed in Django 1.9.
from django.utils import importlib

2016-03-02 01:56:50,530 WARNI [ py.warnings] (proc.22046) <module>():9 /home/omero/OMERO.server/
lib/python/pipeline/utils.py:11: RemovedInDjango19Warning: django.utils.importlib will be removed in Django 1.9.
from django.utils import importlib

2016-03-02 02:15:06,726 WARNI [ py.warnings] (proc.23957) <module>():9 /home/omero/OMERO.server/
lib/python/pipeline/utils.py:11: RemovedInDjango19Warning: django.utils.importlib will be removed in Django 1.9.
from django.utils import importlib

and when pulling up a webpage, I only get a 502 - Bad Gateway page.

I tried restarting nginx. Also tried regenerating the nginx config file but it was almost identical to the one I already had. None of the log entries indicate anything to me why OMERO.web is staying up.

Help please ....
Thanks,
- Damir

[atarkowska]

Hi Damir

Could you try:

Code: Select all

ps aux | grep django.pid

to make sure there are no processes running?

Code: Select all

bin/omero config set omero.web.wsgi_args -- " --error-logfile=/home/omero/OMERO.server/var/log/error.log".

and see if there is anything in this log file?

Could you also give more details about your configuration bin/omero config get --hide-password

Ola

[dsudar]

Hi Ola,

Thanks for the quick response.

Code: Select all

root@quid:~# ps aux | grep django.pid
root     30065  0.0  0.0  11740   948 pts/1    S+   23:18   0:00 grep --color=auto django.pid
root@quid:~#


After setting:

Code: Select all

bin/omero config set omero.web.wsgi_args -- " --error-logfile=/home/omero/OMERO.server/var/log/error.log"

that error.log remains empty:

Code: Select all

omero@quid:~$ cat /home/omero/OMERO.server/var/log/error.log
cat: /home/omero/OMERO.server/var/log/error.log: No such file or directory


Then tried to do "omero web start" but still nothing in that error.log

Code: Select all

omero@quid:~$ omero config get --hide-password
omero.data.dir=/data/OMERO
omero.db.name=omerodb
omero.db.pass=********
omero.db.user=omero
omero.web.application_server=wsgi-tcp
omero.web.apps=["autotag", "tagsearch", "figure"]
omero.web.ui.center_plugins=[["Auto Tag", "autotag/auto_tag_init.js.html", "auto_tag_panel"]]
omero.web.ui.top_links=[["Data", "webindex", {"title": "Browse Data via Projects, Tags etc"}], ["History", "history", {"title": "History"}], ["Help", "http://help.openmicroscopy.org/", {"target": "new", "title": "Open OMERO user guide in a new tab"}], ["Tag Search", "tagsearch"], ["Figure", "figure_index", {"target": "figure", "title": "Open Figure in new tab"}]]
omero@quid:~$


Thanks,
- Damir

[dsudar]

.. and after setting the omero.web.wsgi_args, omero config get reports:

Code: Select all

omero@quid:~$ omero config get --hide-password
omero.data.dir=/data/OMERO
omero.db.name=omerodb
omero.db.pass=********
omero.db.user=omero
omero.web.application_server=wsgi-tcp
omero.web.apps=["autotag", "tagsearch", "figure"]
omero.web.ui.center_plugins=[["Auto Tag", "autotag/auto_tag_init.js.html", "auto_tag_panel"]]
omero.web.ui.top_links=[["Data", "webindex", {"title": "Browse Data via Projects, Tags etc"}], ["History", "history", {"title": "History"}], ["Help", "http://help.openmicroscopy.org/", {"target": "new", "title": "Open OMERO user guide in a new tab"}], ["Tag Search", "tagsearch"], ["Figure", "figure_index", {"target": "figure", "title": "Open Figure in new tab"}]]
omero.web.wsgi_args= --error-logfile=/home/omero/OMERO.server/var/log/error.log


But still nothing in the that error.log

I just looked it up and noticed that that relates to gunicorn. And gunicorn is not running:

Code: Select all

root@quid:~# ps aux | grep gunicorn
root     30672  0.0  0.0  11740   940 pts/1    S+   23:50   0:00 grep --color=auto gunicorn


And the rest of OMERO is perfectly fine. I can connect with Insight with no trouble.

Thanks,
- Damir

[atarkowska]

Could you join us on IRC, either via IRC client or https://webchat.freenode.net/, channel #ome. I will try to help you directly.

Ola

[atarkowska]

Damir

Sorry I haven't realized the timezone difference . If you cannot make to IRC early morning, then

1. I understand that you tried:

Code: Select all

/home/omero/OMERO.server/bin/omero config set omero.web.wsgi_args -- ' --error-logfile=/home/omero/OMERO.server/var/log/error.log'
/home/omero/OMERO.server/bin/omero web start

then

Code: Select all

    root@quid:~# ps aux | grep gunicorn
    root     30672  0.0  0.0  11740   940 pts/1    S+   23:50   0:00 grep --color=auto gunicorn

no web workers are running and

Code: Select all

    omero@quid:~$ cat /home/omero/OMERO.server/var/log/error.log
    cat: /home/omero/OMERO.server/var/log/error.log: No such file or directory

correct?
Coudl you try to change log level

Code: Select all

cd /home/omero/OMERO.server/
$ bin/omero config set omero.web.wsgi_args -- ' --log-level=DEBUG --error-logfile=/home/omero/OMERO.server/var/log/error.log'
$ bin/omero web start

Could you paste the full output of bin/omero web start?

I noticed you are running some of these commands as root and some as omero user. Please make sure you are starting omeroweb as omero user.

2. As the above didn't help, could you try

Code: Select all

$ export PYTHONPATH=/home/omero/OMERO.server/lib/python:$PYTHONPATH
$ gunicorn --bind 127.0.0.1:4080  omeroweb.wsgi:application

does this give any error?

Ola

[dsudar]

All,

I just had a support session with Ola who figured out the problem:
Ubuntu 14.04.4's default gunicorn version is 17.5 but OMERO requires 19.3+.

So we forced an upgrade with pip:

Code: Select all

root@quid:/home/omero/OMERO.server# pip install --upgrade -r share/web/requirements-py27-nginx.txt

and that upgraded it to the compatible versions of gunicorn and django.

Now works fine.
Thanks Ola!

- Damir