Bad Gateway

[phm]

Hi,

I'm using omero web from nginx, everything work nice except from today (I have "bad gateway" error when I try to reach my server. I have tested that omero web is running :
omero web status
OMERO.web status... [RUNNING] (PID 5716)

If I try to connect to the root of the web server I got the Welcome message from Nginx. It seems that the problem come from the omero web server, but I can't identify the source.
The connection via insight is working....

Any ideas ?

Philippe

I have searched to understand while omero web is down. And it seems that the problem come after an upgrade of php-fpm which is used by nginx to run php requests. However, I can't find any solution. Is there a reason to use nginx versus apache except for the efficiency to treated dynamic requests. Because may be if I configure OMERO to use apache instead of ngnix the problem disappear .

[atarkowska]

Hi Philippe,

To be able to identify what is exactly the problem, could you:
- send your omero nginx conf,
- send what is in the nginx logfile when you access the URL
- send output of your bin/omero config get

If this is due to the upgrade of php-fpm what exactly nginx log says ?

Ola

[phm]

Hi Philippe,

To be able to identify what is exactly the problem, could you:
- send your omero nginx conf,
- send what is in the nginx logfile when you access the URL
- send output of your bin/omero config get

If this is due to the upgrade of php-fpm what exactly nginx log says ?

Ola

Hi Ola,

Thanks for the answer. First I didn't change the conf files either for OMERO or Nginx. And I have implemented the HRM web interface to do Huygens devolutions on the OMERO server and it's work.
However, here the conf files :
Nginx omero conf
server {
listen 80 default;
server_name omero.college-de-france.fr;
rewrite ^ https://$server_name$request_uri? permanent;
}
server {
listen 443 default ssl;
server_name omero.college-de-france.fr;

ssl on;
ssl_certificate /etc/nginx/omero.college-de-france.fr.crt;
ssl_certificate_key /etc/nginx/omero.college-de-france.fr.key;

ssl_session_timeout 5m;

ssl_protocols SSLv2 SSLv3 TLSv1;
ssl_ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
ssl_prefer_server_ciphers on;

# weblitz django apps serve media from here
location /static {
alias /opt/OMERO.server/lib/python/omeroweb/static;
}

location /omero {
if (-f /opt/OMERO.server/var/maintenance.html) {
error_page 503 /maintenance.html;
return 503;
}

fastcgi_pass 0.0.0.0:4080;

fastcgi_split_path_info ^(/omero)(.*)$;
fastcgi_param PATH_INFO $fastcgi_path_info;
fastcgi_param SCRIPT_INFO $fastcgi_script_name;


fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param SERVER_NAME $server_name;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param SERVER_PORT $server_port;
fastcgi_pass_header Authorization;
fastcgi_intercept_errors on;
fastcgi_read_timeout 300;
# Uncomment if nginx SSL module is enabled or you are using nginx 1.1.11 or later
# -- See: #10273, http://nginx.org/en/CHANGES
fastcgi_param HTTPS $https;
}

location /maintenance.html {
root /opt/OMERO.server/var;
}
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}

}

nginx access.log when I try to load the web omero page with Bad gateway

192.168.188.18 - - [09/Dec/2014:17:48:08 +0100] "GET /omero/webclient/login/?url=%2Fomero%2Fwebclient%2F HTTP/1.1" 502 574 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36" "-"

nginx error.log

2014/12/09 17:51:39 [crit] 33674#0: *219 stat() "/opt/OMERO.server/var/maintenance.html" failed (13: Permission denied), client: 192.168.188.18, server: omero.college-de-france.fr, request: "GET /omero/webclient/login/?url=%2Fomero%2Fwebclient%2F HTTP/1.1", host: "omero.college-de-france.fr"
2014/12/09 17:51:39 [crit] 33674#0: *219 connect() to 0.0.0.0:4080 failed (13: Permission denied) while connecting to upstream, client: 192.168.188.18, server: omero.college-de-france.fr, request: "GET /omero/webclient/login/?url=%2Fomero%2Fwebclient%2F HTTP/1.1", upstream: "fastcgi://0.0.0.0:4080", host: "omero.college-de-france.fr"

omero config get

omero.data.dir=/OMERO
omero.db.name=omero_database
omero.db.pass=**************************************
omero.db.user=db_omero
omero.web.admins=["Philippe", "**************@**********"]
omero.web.application_server=fastcgi-tcp
omero.web.apps=["figure"]
omero.web.force_script_name=/omero
omero.web.login_logo=/static/webclient/image/logo_CIRB.jpg
omero.web.server_email=****************@******************
omero.web.ui.top_links=[["Figure", "figure_index", {"target": "figure"}]]
[omero@omero ~]$ omero config edit
[omero@omero ~]$ omero config get
omero.data.dir=/OMERO
omero.db.name=omero_database
omero.db.pass=********************************
omero.db.user=db_omero
omero.web.admins=["Philippe", "*************@******************"]
omero.web.application_server=fastcgi-tcp
omero.web.apps=["figure"]
omero.web.force_script_name=/omero
omero.web.login_logo=/static/webclient/image/logo_CIRB.jpg
omero.web.server_email=************@*************
omero.web.ui.top_links=[["Figure", "figure_index", {"target": "figure"}]]


php-fpm www-error.log is empty

and php-fpm error.log

[09-déc.-2014 12:29:46] NOTICE: fpm is running, pid 14611
[09-déc.-2014 12:29:46] NOTICE: ready to handle connections
[09-déc.-2014 12:39:27] NOTICE: Terminating ...
[09-déc.-2014 12:39:27] NOTICE: exiting, bye-bye!
[09-déc.-2014 12:39:27] NOTICE: fpm is running, pid 15049
[09-déc.-2014 12:39:27] NOTICE: ready to handle connections
[09-déc.-2014 12:40:52] NOTICE: Terminating ...
[09-déc.-2014 12:40:52] NOTICE: exiting, bye-bye!
[09-déc.-2014 12:40:52] NOTICE: fpm is running, pid 15143
[09-déc.-2014 12:40:52] NOTICE: ready to handle connections



Philippe

[phm]

Hi Ola,

It's seem that the problem come from permission access, the nginx error.log give these 2 errors then I try to connect to the web side of the server :
I don't understand the first error message, I didn't have any maintenance.html file in /opt/OMERO.server/var/ ???

2014/12/10 10:44:11 [crit] 33672#0: *2391 stat() "/opt/OMERO.server/var/maintenance.html" failed (13: Permission denied), client: 192.168.188.18, server: omero.college-de-france.fr, request: "GET /omero/webclient/login/?url=%2Fomero%2Fwebclient%2F HTTP/1.1", host: "omero.college-de-france.fr"
2014/12/10 10:44:11 [crit] 33672#0: *2391 connect() to 0.0.0.0:4080 failed (13: Permission denied) while connecting to upstream, client: 192.168.188.18, server: omero.college-de-france.fr, request: "GET /omero/webclient/login/?url=%2Fomero%2Fwebclient%2F HTTP/1.1", upstream: "fastcgi://0.0.0.0:4080", host: "omero.college-de-france.fr"

Philippe

[atarkowska]

Hi Philippe

2014/12/09 17:51:39 [crit] 33674#0: *219 stat() "/opt/OMERO.server/var/maintenance.html" failed (13: Permission denied), client: 192.168.188.18, server: omero.college-de-france.fr, request: "GET /omero/webclient/login/?url=%2Fomero%2Fwebclient%2F HTTP/1.1", host: "omero.college-de-france.fr"

is accessed when your upstream application is gives HTTP 503. This is in your nginx.conf

Code: Select all

if (-f /opt/OMERO.server/var/maintenance.html) {
error_page 503 /maintenance.html;
return 503;
}
...
location /maintenance.html {
root /opt/OMERO.server/var;
}


Usually when upstream app is down (61: Connection refused) that page can be added to tell users about maintenance.

I am not sure how php-fpm installation could affect webclient but did you try to restart web and nginx:

Code: Select all

bin/omero web stop
restart nginx
bin/omero web start

?

Could you also check if permissions on the omero directories hasn't change?
If that is permissions issue you may try play with chmod -R 775 on the OMERO dir

Which OS are you running on?

I understand that if you access http://omero.college-de-france.fr index page is displayed correctly?
Ola

[phm]

Hi Philippe

2014/12/09 17:51:39 [crit] 33674#0: *219 stat() "/opt/OMERO.server/var/maintenance.html" failed (13: Permission denied), client: 192.168.188.18, server: omero.college-de-france.fr, request: "GET /omero/webclient/login/?url=%2Fomero%2Fwebclient%2F HTTP/1.1", host: "omero.college-de-france.fr"

is accessed when your upstream application is gives HTTP 503. This is in your nginx.conf

Code: Select all

if (-f /opt/OMERO.server/var/maintenance.html) {
error_page 503 /maintenance.html;
return 503;
}

Ok I see.

I am not sure how php-fpm installation could affect webclient :

In fact I use php-fpm with the virtual host corresponding to the HRM web client it use fastcgi_pass and t ok !

#
# Another server, for Hyugens remote manager.
#

server {
listen 8080;
server_name omero-web.college-de-france.fr;
root /var/www/html/hrm;
index index.php index.html index.htm;

#access_log logs/host.access.log main;

location / {
try_files $uri $uri/ /index.php;
}



# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
location ~ \.php$ {

try_files $uri =404;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
}

If I change the fastcgi_pass in omero.conf from 0.0.0.0:4080 to 127.0.01:9000
I didn't get the bad gateway error but of course the web page is blank !

but did you try to restart web and nginx

Code: Select all

bin/omero web stop
restart nginx
bin/omero web start

I did but same bad gateway.

Could you also check if permissions on the omero directories hasn't change?
If that is permissions issue you may try play with chmod -R 775 on the OMERO dir

No change

Which OS are you running on?
Ola

I using CentOS 6.5. And before upgrading everything was working.


Philippe

[atarkowska]

Hi Philippe,

In fact I use php-fpm with the virtual host corresponding to the HRM web client it use fastcgi_pass and t ok !

#
# Another server, for Hyugens remote manager.
#

server {
listen 8080;
server_name omero-web.college-de-france.fr;
root /var/www/html/hrm;
index index.php index.html index.htm;

#access_log logs/host.access.log main;

location / {
try_files $uri $uri/ /index.php;
}



# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
location ~ \.php$ {

try_files $uri =404;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
}

Could you disable php-fpm and see if omero.web is back to normal?

If I change the fastcgi_pass in omero.conf from 0.0.0.0:4080 to 127.0.01:9000
I didn't get the bad gateway error but of course the web page is blank !

You can't change port. Check how your workers are run

Code: Select all

$ ps aux | grep django.pid

omero-x  10057  0.0  0.0 280132  9412 ?        S    Nov12   0:16 python manage.py runfcgi workdir=./ method=prefork host=0.0.0.0 port=4080 pidfile=.../var/django.pid daemonize=true maxchildren=5 minspare=1 maxspare=5 maxrequests=400
... and 5 more


OMERO.web workers by default are listening on 0.0.0.0 on on port 4080. Then fastcgi_pass 0.0.0.0:4080 should work though.

Ola

[phm]

Hi Philippe,


Could you disable php-fpm and see if omero.web is back to normal?

Hi Ola

When I stop php-fpm omero.web and hrm display the bad gateway.

You can't change port. Check how your workers are run

Code: Select all

$ ps aux | grep django.pid

omero-x  10057  0.0  0.0 280132  9412 ?        S    Nov12   0:16 python manage.py runfcgi workdir=./ method=prefork host=0.0.0.0 port=4080 pidfile=.../var/django.pid daemonize=true maxchildren=5 minspare=1 maxspare=5 maxrequests=400
... and 5 more


OMERO.web workers by default are listening on 0.0.0.0 on on port 4080. Then fastcgi_pass 0.0.0.0:4080 should work though.

Ola

Yes I have the same workers :

ps aux | grep django.pid
omero 20985 0.0 0.1 289952 55132 ? S 11:57 0:00 python manage.py runfcgi workdir=./ method=prefork host=0.0.0.0 port=4080 pidfile=/opt/OMERO.server/var/django.pid daemonize=true maxchildren=5 minspare=1 maxspare=5 maxrequests=400
omero 20986 0.0 0.1 289952 54736 ? S 11:57 0:00 python manage.py runfcgi workdir=./ method=prefork host=0.0.0.0 port=4080 pidfile=/opt/OMERO.server/var/django.pid daemonize=true maxchildren=5 minspare=1 maxspare=5 maxrequests=400
omero 20987 0.0 0.1 289952 54740 ? S 11:57 0:00 python manage.py runfcgi workdir=./ method=prefork host=0.0.0.0 port=4080 pidfile=/opt/OMERO.server/var/django.pid daemonize=true maxchildren=5 minspare=1 maxspare=5 maxrequests=400
omero 20988 0.0 0.1 289952 54740 ? S 11:57 0:00 python manage.py runfcgi workdir=./ method=prefork host=0.0.0.0 port=4080 pidfile=/opt/OMERO.server/var/django.pid daemonize=true maxchildren=5 minspare=1 maxspare=5 maxrequests=400
omero 20989 0.0 0.1 289952 54740 ? S 11:57 0:00 python manage.py runfcgi workdir=./ method=prefork host=0.0.0.0 port=4080 pidfile=/opt/OMERO.server/var/django.pid daemonize=true maxchildren=5 minspare=1 maxspare=5 maxrequests=400
omero 20990 0.0 0.1 289952 54740 ? S 11:57 0:00 python manage.py runfcgi workdir=./ method=prefork host=0.0.0.0 port=4080 pidfile=/opt/OMERO.server/var/django.pid daemonize=true maxchildren=5 minspare=1 maxspare=5 maxrequests=400
root 26633 0.0 0.0 105324 952 pts/0 S+ 14:41 0:00 grep django.pid

I also tried to disable hrm conf (may be conflicts with omero.web) but no change.
Is it possible to change the port and the host for django on 127.0.0.1:9000 (if that makes sense)

Philippe

[manics]

Is it possible to change the port and the host for django on 127.0.0.1:9000 (if that makes sense)

Hi Philippe, as Ola mentioned port 4080 is only used internally for communication between nginx and OMERO.web, so changing it is unlikely to help. Could you try returning your Nginx configuration to it's original state when first installed (so delete any configuration related to php-fpm, HRM, or anything else), stop everything (OMERO, OMERO.web, nginx), verify that OMERO.web is stopped (ps aux | grep django.pid), and start everything again.

Simon

[phm]

Is it possible to change the port and the host for django on 127.0.0.1:9000 (if that makes sense)

Hi Philippe, as Ola mentioned port 4080 is only used internally for communication between nginx and OMERO.web, so changing it is unlikely to help. Could you try returning your Nginx configuration to it's original state when first installed (so delete any configuration related to php-fpm, HRM, or anything else), stop everything (OMERO, OMERO.web, nginx), verify that OMERO.web is stopped (ps aux | grep django.pid), and start everything again.

Simon

Hi Simon,
I did before some of your suggestions (disable all conf related to nginx) , but I didn't stop omero server because I have users on it. However, I could try that this week-end "fortunately" we will get a power shutdown !!!
I will report you the results next week.

Philippe