Omero don't start after upgrade

[kennethgillen]

Is there anything more in the error message?

I'd be looking to find out what the NGINX logs are saying at the same time as these errors:

e.g.

System nginx logs (CentOS/RHEL e.g.) /var/log/nginx/{access.log,error.log}

I didn't see any of those in the QA upload?

Cheers,

Kenny

[kennethgillen]

I also wonder what the systemd unit file looks like, out of curiosity?

Code: Select all

cat /etc/systemd/system/omero-web.service

[phm]

Is there anything more in the error message?

I'd be looking to find out what the NGINX logs are saying at the same time as these errors:

e.g.

System nginx logs (CentOS/RHEL e.g.) /var/log/nginx/{access.log,error.log}

I didn't see any of those in the QA upload?

Cheers,

Kenny

nginx error.log :

Code: Select all

omero@omero:~$>sudo more /var/log/nginx/error.log
2017/12/12 14:20:51 [error] 6934#0: *4977 open() "/usr/share/nginx/html/static/webgateway/css/reset.css" failed (2: No such file or directory), client: 192.1
68.188.18, server: omero.college-de-france.fr, request: "GET /static/webgateway/css/reset.css?_5.4.1-ice36-b75 HTTP/1.1", host: "omero.college-de-france.fr",
referrer: "https://omero.college-de-france.fr/omero/webclient/login/?url=%2Fomero%2Fwebclient%2F"
2017/12/12 14:20:51 [error] 6934#0: *4977 open() "/usr/share/nginx/html/static/webgateway/css/ome.body.css" failed (2: No such file or directory), client: 19
2.168.188.18, server: omero.college-de-france.fr, request: "GET /static/webgateway/css/ome.body.css?_5.4.1-ice36-b75 HTTP/1.1", host: "omero.college-de-franc
e.fr", referrer: "https://omero.college-de-france.fr/omero/webclient/login/?url=%2Fomero%2Fwebclient%2F"
2017/12/12 14:20:51 [error] 6934#0: *4977 open() "/usr/share/nginx/html/static/webclient/css/dusty.css" failed (2: No such file or directory), client: 192.16
8.188.18, server: omero.college-de-france.fr, request: "GET /static/webclient/css/dusty.css?_5.4.1-ice36-b75 HTTP/1.1", host: "omero.college-de-france.fr", r
eferrer: "https://omero.college-de-france.fr/omero/webclient/login/?url=%2Fomero%2Fwebclient%2F"
2017/12/12 14:20:51 [error] 6934#0: *4979 open() "/usr/share/nginx/html/static/feedback/css/layout.css" failed (2: No such file or directory), client: 192.16
8.188.18, server: omero.college-de-france.fr, request: "GET /static/feedback/css/layout.css HTTP/1.1", host: "omero.college-de-france.fr", referrer: "https:/
/omero.college-de-france.fr/omero/webclient/login/?url=%2Fomero%2Fwebclient%2F"
2017/12/12 14:20:51 [error] 6934#0: *4979 open() "/usr/share/nginx/html/static/webgateway/img/ome.ico" failed (2: No such file or directory), client: 192.168
.188.18, server: omero.college-de-france.fr, request: "GET /static/webgateway/img/ome.ico HTTP/1.1", host: "omero.college-de-france.fr", referrer: "https://o
mero.college-de-france.fr/omero/webclient/login/?url=%2Fomero%2Fwebclient%2F"



Philippe

[phm]

I also wonder what the systemd unit file looks like, out of curiosity?

Code: Select all

cat /etc/systemd/system/omero-web.service

Code: Select all

omero@omero:~$>cat /etc/systemd/system/omero-web.service
[Unit]
Description=OMERO.web
# Not mandatory, Nginx may be running on a different server
Requires=nginx.service
After=network.service

[Service]
User=omero
Type=forking
PIDFile=/home/omero/OMERO.py/var/django.pid
Restart=no
RestartSec=10
Environment="PATH=/home/omero/omerowebvenv/bin:/bin:/usr/bin"
Environment="WEBBINDIR=/home/omero/OMERO.py/bin"
ExecStart=/home/omero/omerowebvenv/bin/python /home/omero/OMERO.py/bin/omero web start
ExecStop=/home/omero/omerowebvenv/bin/python /home/omero/OMERO.py/bin/omero web stop

[Install]
WantedBy=multi-user.target



Philippe

[kennethgillen]

"/usr/share/nginx/html/static/webgateway/css/reset.css" failed (2: No such file or directory), client: 192.1

This tells us NGINX is trying to read the wrong filesystem location for the statics.

Since your NGINX config has now changed, could you please paste the current versions of: your

• full NGINX omero_ngnix.conf
• the higher-level /etc/nginx/nginx.conf ?

[phm]

"/usr/share/nginx/html/static/webgateway/css/reset.css" failed (2: No such file or directory), client: 192.1

This tells us NGINX is trying to read the wrong filesystem location for the statics.

Since your NGINX config has now changed, could you please paste the current versions of: your

• full NGINX omero_ngnix.conf
• the higher-level /etc/nginx/nginx.conf ?

Code: Select all

omero@omero:~$>sudo more /etc/nginx/nginx.conf
[sudo] Mot de passe de omero : 
# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80; #; #; # default_server;
        listen       [::]:80; #; #; # default_server;
        server_name  _;
        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }

# Settings for a TLS enabled server.
#
#    server {
#        listen       443 ssl http2; #; #; # default_server;
#        listen       [::]:443 ssl http2; #; #; # default_server;
#        server_name  _;
#        root         /usr/share/nginx/html;
#
#        ssl_certificate "/etc/pki/nginx/server.crt";
#        ssl_certificate_key "/etc/pki/nginx/private/server.key";
#        ssl_session_cache shared:SSL:1m;
#        ssl_session_timeout  10m;
#        ssl_ciphers HIGH:!aNULL:!MD5;
#        ssl_prefer_server_ciphers on;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
#        location / {
#        }
#
#        error_page 404 /404.html;
#            location = /40x.html {
#        }
#
#        error_page 500 502 503 504 /50x.html;
#            location = /50x.html {
#        }
#    }

}



Code: Select all

omero@omero:~$>sudo more /etc/nginx/conf.d/omero.conf
upstream omeroweb {
    server 127.0.0.1:4080 fail_timeout=0;
}

server {
    listen 80;
    server_name localhost;

    sendfile on;
    client_max_body_size 0;

    # maintenance page serve from here
    location @maintenance {
        root /home/omero/OMERO.py/etc/templates/error;
        try_files $uri /maintainance.html =502;
    }

    # weblitz django apps serve media from here
    location /static {
        alias /home/omero/OMERO.py/lib/python/omeroweb/static;
    }

    location @proxy_to_app {
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_buffering off;

        proxy_pass http://omeroweb;
    }

    location / {

        error_page 502 @maintenance;
        # checks for static file, if not found proxy to app
        try_files $uri @proxy_to_app;
    }

}



Philippe

[kennethgillen]

Hi Philippe,

That's looks like the problem: you've a top-level NGINX config overriding OMERO's.

The top-level /etc/nginx/nginx.conf has a server definition which is taking precedence.

I would totally remove that, and the omero NGINX config should start to take effect.

i.e.

/etc/nginx/nginx.conf

Code: Select all

server {
...
}

*note: this would have to have been introduced at some point after OMERO.web last worked, but before NGINX was reloaded during the upgrade - so it's worth checking where in your upgrade process this file was touched.

Please let us know how you get on.

[phm]

Hi Philippe,

That's looks like the problem: you've a top-level NGINX config overriding OMERO's.

The top-level /etc/nginx/nginx.conf has a server definition which is taking precedence.

I would totally remove that, and the omero NGINX config should start to take effect.

i.e.

/etc/nginx/nginx.conf

Code: Select all

server {
...
}

*note: this would have to have been introduced at some point after OMERO.web last worked, but before NGINX was reloaded during the upgrade - so it's worth checking where in your upgrade process this file was touched.

Please let us know how you get on.

I remove the server info in the nginx.conf, however that don't change ???

Code: Select all

omero@omero:~$>sudo more /etc/nginx/nginx.conf
# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

#    server {
#        listen       80; #; #; # default_server;
#        listen       [::]:80; #; #; # default_server;
#        server_name  _;
#        root         /usr/share/nginx/html;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
#        location / {
#        }
#
#        error_page 404 /404.html;
#            location = /40x.html {
#        }
#
#        error_page 500 502 503 504 /50x.html;
#            location = /50x.html {
#        }
#    }

# Settings for a TLS enabled server.
#
#    server {
#        listen       443 ssl http2; #; #; # default_server;
#        listen       [::]:443 ssl http2; #; #; # default_server;
#        server_name  _;
#        root         /usr/share/nginx/html;
#
#        ssl_certificate "/etc/pki/nginx/server.crt";
#        ssl_certificate_key "/etc/pki/nginx/private/server.key";
#        ssl_session_cache shared:SSL:1m;
#        ssl_session_timeout  10m;
#        ssl_ciphers HIGH:!aNULL:!MD5;
#        ssl_prefer_server_ciphers on;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
#        location / {
#        }
#
#        error_page 404 /404.html;
#            location = /40x.html {
#        }
#
#        error_page 500 502 503 504 /50x.html;
#            location = /50x.html {
#        }
#    }

}



Code: Select all

omero@omero:~$>sudo systemctl restart nginx
omero@omero:~$>service omero-web.service stop
omero@omero:~$>service omero-web.service start
Redirecting to /bin/systemctl start omero-web.service


Philippe

[kennethgillen]

How about other NGINX configuration files in /etc/nginx/conf.d/ ?

Code: Select all

ls -alhrt  /etc/nginx/conf.d/

If you have any other files (like a default) also specifying Server blocks, then they may also be clashing.

[phm]

How about other NGINX configuration files in /etc/nginx/conf.d/ ?

Code: Select all

ls -alhrt  /etc/nginx/conf.d/

If you have any other files (like a default) also specifying Server blocks, then they may also be clashing.

Code: Select all

omero@omero:~$>ls -alhrt  /etc/nginx/conf.d/
total 16K
-rw-r--r--  1 root  root   984 23 août  17:06 omeroweb.conf.old
-rw-rw-r--  1 omero omero  937 11 déc.  17:54 omero.conf
drwxr-xr-x. 2 root  root  4,0K 12 déc.  15:22 .
drwxr-xr-x. 4 root  root  4,0K 12 déc.  18:08 ..


Code: Select all

omero@omero:~$>ls -alhrt  /etc/nginx/
total 96K
-rw-r--r--   1 root root 2,5K 18 août  13:03 nginx.conf.bak
-rw-r--r--   1 root root 1,7K 23 août  17:15 omero.college-de-france.fr.key
-rw-r--r--   1 root root 3,7K 23 août  17:41 omero.college-de-france.fr.pem
-rw-r--r--   1 root root 3,6K 18 sept. 11:20 win-utf
-rw-r--r--   1 root root  664 18 sept. 11:20 uwsgi_params.default
-rw-r--r--   1 root root  664 18 sept. 11:20 uwsgi_params
-rw-r--r--   1 root root  636 18 sept. 11:20 scgi_params.default
-rw-r--r--   1 root root  636 18 sept. 11:20 scgi_params
-rw-r--r--   1 root root 2,6K 18 sept. 11:20 nginx.conf.default
-rw-r--r--   1 root root 3,9K 18 sept. 11:20 mime.types.default
-rw-r--r--   1 root root 3,9K 18 sept. 11:20 mime.types
-rw-r--r--   1 root root 2,2K 18 sept. 11:20 koi-win
-rw-r--r--   1 root root 2,8K 18 sept. 11:20 koi-utf
-rw-r--r--   1 root root 1007 18 sept. 11:20 fastcgi_params.default
-rw-r--r--   1 root root 1007 18 sept. 11:20 fastcgi_params
-rw-r--r--   1 root root 1,1K 18 sept. 11:20 fastcgi.conf.default
-rw-r--r--   1 root root 1,1K 18 sept. 11:20 fastcgi.conf
drwxr-xr-x.  2 root root 4,0K 18 sept. 11:22 default.d
drwxr-xr-x. 92 root root  12K 12 déc.  10:36 ..
drwxr-xr-x.  2 root root 4,0K 12 déc.  15:22 conf.d
-rw-r--r--   1 root root 2,5K 12 déc.  18:08 nginx.conf
drwxr-xr-x.  4 root root 4,0K 12 déc.  18:08 .


Code: Select all

omero@omero:~$>ls -alhrt  /etc/nginx/default.d/
total 8,0K
drwxr-xr-x. 2 root root 4,0K 18 sept. 11:22 .
drwxr-xr-x. 4 root root 4,0K 12 déc.  18:08 ..



Philippe