We're Hiring!

"tmp" running full

Having a problem deploying OMERO? Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

The OMERO.server installation documentation begins here and you can find OMERO.web deployment documentation here.

"tmp" running full

Postby ehrenfeu » Wed Oct 16, 2013 11:38 am

Dear all,

recently one of our users noticed a problem when trying to create a movie export using the "publishing options" -> "make movie" function. However, he just got an error message saying there was "no processor" available.

Digging around on the machine, I figured that the "tmp" directory (<OMEROroot>/omero/tmp/) contains huge amounts of big files (files named "__omero_export__[0-9]{18}.ome.tif"), filling up the partition where the directory resides. In our case that's a separate one, containing nothing but the "omero/tmp/" and "var/log/" directories of OMERO (*not* the system's ones).

In addition, I found that the "processor" Python process ate up all the memory available on that machine (32GB overall):
Code: Select all
$ ps fu -e | grep runProcessor
omeronas  3764  2.1 93.6 34488308 30774112 ?   Sl   Sep27 582:55  \_ python lib/python/runProcessor.py --Ice.Config=/home/omeronas/OMERO.server-4.4.5-ice34-b20/var/master/servers/Processor-0/config/config


So, where does all the data in the "tmp" directory come from, and what happened that prevented the directory from being cleaned up? Clearly, most of the stuff was placed there yesterday when our user tried to create the movies, but there is also some stuff sitting there already since march:

Code: Select all
du --time -sh tmp/*
3,2G 2013-03-26 11:49   14980@bc2-omero01
245M 2013-09-11 17:20   28414@bc2-omero01.example.com
148K 2013-09-17 15:50   28448
20G  2013-10-15 14:05   3734@bc2-omero01.example.com
16G  2013-10-15 15:10   3764


Is the behaviour of the Python process expected, or is this rather a bug that was uncovered by running out of space in the tmp dir?

And last (but most important): how should we recover right now? I'd just try to stop the OMERO services as usual making sure the berserk python process is gone, (re)move the ome.tiffs from the "tmp" directory and then start the services again.

Many thanks,
Niko
User avatar
ehrenfeu
 
Posts: 90
Joined: Fri May 11, 2012 8:21 am
Location: Basel, Switzerland

Re: "tmp" running full

Postby sbesson » Thu Oct 17, 2013 9:29 am

Hi Niko,

the `tmp` bug you are describing above has been ticketed https://trac.openmicroscopy.org.uk/ome/ticket/11074 and fixed in https://github.com/openmicroscopy/openm ... /pull/1261. The latter fix should be included in the 4.4.9 release that came out yesterday.

Hopefully, upgrading your server to 4.4.9 should be all that is needed to fully resolve this issue.

Let us know if you are still experiencing problems,
Sebastien
User avatar
sbesson
Team Member
 
Posts: 421
Joined: Tue Feb 28, 2012 7:20 pm

Re: "tmp" running full

Postby ehrenfeu » Fri Oct 18, 2013 8:40 am

Hi Sébastien,

thanks for this information, we'll do the upgrade soon then :)

Cheers
Niko
User avatar
ehrenfeu
 
Posts: 90
Joined: Fri May 11, 2012 8:21 am
Location: Basel, Switzerland

Re: "tmp" running full

Postby jmoore » Wed Oct 23, 2013 11:31 am

Niko,

also, while the server is down for upgrading, delete all of the temp directory to more quickly recover that space. If you need to point the temp directory at another location to prevent future issues, see https://www.openmicroscopy.org/site/support/omero4/developers/Modules/TempFileManager.html

Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany


Return to Installation and Deployment

Who is online

Users browsing this forum: No registered users and 1 guest