We're Hiring!

'Failed to shutdown some components'

Having a problem deploying OMERO? Here's a great place to start.
Please note:
The OMERO.server installation documentation begins here and you can find OMERO.web deployment documentation here.

'Failed to shutdown some components'

Postby achessel » Wed Jan 11, 2017 2:52 pm

Hi all,

Following a brief 'disk full' event on the disk the omero server is running out of, the server is in a weird state. But it does not seem to want to be shutdown, saying it can't reach node 'master'.

Is there a way to force it down cleanishly, or can I just kill the process? Anything I should do before I start it up again if I just kiil it, or can I expect some things to be broken? (I don't think anything was going on when the original error happened)

Thanks
A.
achessel
 
Posts: 67
Joined: Fri Jan 14, 2011 1:58 pm

Re: 'Failed to shutdown some components'

Postby mtbc » Thu Jan 12, 2017 9:47 am

"can't reach node 'master'" is a new one for us I'm afraid. I'd recommend simply killing the icebox and icegridnode processes with SIGTERM (give them some seconds before resorting to SIGKILL) then once you are sure the processes have gone then removing the .lock files from inside your binary repository, e.g., probably those reported by, find /OMERO -name .lock

I think you should then expect to be able to start up the server processes just fine. Still, weirdness can happen when one runs out of space, so the cautious may wish to back up before restarting the server, and if you do see any oddness, or require any elaboration of the above, then please don't hesitate to ask.

Cheers, and good luck,
Mark
User avatar
mtbc
Team Member
 
Posts: 119
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Re: 'Failed to shutdown some components'

Postby kennethgillen » Thu Jan 12, 2017 10:19 am

achessel wrote:Following a brief 'disk full' event on the disk the omero server is running out of, the server is in a weird state. But it does not seem to want to be shutdown, saying it can't reach node 'master'.


I'd also recommend adding some monitoring to your server, something like Check_MK, Munin, or some other system of which there are plenty to choose from. [1]

OME have experience of monitoring OMERO servers with Check_MK amd Munin, and others in the community may well have experience with other tools.

[1] https://github.com/kahun/awesome-sysadmin#monitoring

All the best,

Kenny
User avatar
kennethgillen
Team Member
 
Posts: 175
Joined: Mon Nov 05, 2012 3:39 pm

Re: 'Failed to shutdown some components'

Postby achessel » Fri Jan 13, 2017 3:47 pm

Thanks for the info.
I sent SIGTERM to icegridnode, which killed everything omero related except icegridnode (?), so I then had to SIGKILL it.
But server is back up and seem fine. It would definitely help to have more monitoring, I'll look it up. But we currently are functioning without a dedicated IT person in the lab which is not helping...
achessel
 
Posts: 67
Joined: Fri Jan 14, 2011 1:58 pm


Return to Installation and Deployment

Who is online

Users browsing this forum: Google [Bot] and 2 guests