We're Hiring!

large number of temporay files - is this normal

Having a problem deploying OMERO? Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

The OMERO.server installation documentation begins here and you can find OMERO.web deployment documentation here.

large number of temporay files - is this normal

Postby bernie » Tue Mar 18, 2014 1:01 pm

Hi,

we've just moved OMERO to new hardware. All our data files are stored on the University SAN and the new server is pointing to these. The server has been built in close collaboration with the University IT department and is under Puppet management. Currently the SAN shows that there is 800G free out of a total of 8.8TB (92% utilisation). Nevertheless we cannot right to the SAN; we get the error: No space left on device. If you list open files though you can see that all available space is filed with temporary files help open by JAVA processes associated with OMERO (stop OMERO and they disappear) (see attachment). This wasn't a problem with the previous server so I'm wondering if this is normal? I've also attached the server diagnostics.
Attachments
diagnostics.zip
(2.49 KiB) Downloaded 140 times
bernie
 
Posts: 50
Joined: Mon Aug 17, 2009 12:03 pm

Re: large number of temporay files - is this normal

Postby manics » Wed Mar 19, 2014 12:13 pm

Hi

You've only got 118 files open which looks normal and should be well below the OS limit. The files under FullText are related to search indexing. Could the inability to write to the free space be due to storage restrictions, e.g. some space being reserved for admin purposes only?

What else is being stored on the SAN? Can you write to it outside of OMERO (but as the OMERO user?).

Simon
User avatar
manics
Team Member
 
Posts: 261
Joined: Mon Oct 08, 2012 11:01 am
Location: Dundee

Re: large number of temporay files - is this normal

Postby bernie » Wed Mar 19, 2014 3:24 pm

Hey Simon,

well I think the IT dept is retreating from their original position and accepting there might be a problem with the SAN or how the new server is connecting to the SAN. Cannot write to the SAN when the services aren't running. The SAN stores the campus! but seems to be running normally otherwise. It's reassuring to know that files that are open are as expected. We also had to change the omero id on the new server and that now corresponds to the old omero id. I shall update this post when I know more,

Bernie
bernie
 
Posts: 50
Joined: Mon Aug 17, 2009 12:03 pm

Re: large number of temporay files - is this normal

Postby manics » Wed Mar 19, 2014 4:08 pm

Hi Bernie

A colleague just pointed out that it could be another partition that's filled up (e.g. a temp directory). What does df -h show? Presumably /OMERO is a symlink to your SAN? If you still have problems could you give us your server logs, either as an attachment here or upload them to https://www.openmicroscopy.org/qa2/qa/upload/

Cheers

Simon
User avatar
manics
Team Member
 
Posts: 261
Joined: Mon Oct 08, 2012 11:01 am
Location: Dundee

Re: large number of temporay files - is this normal

Postby bernie » Wed Mar 19, 2014 5:44 pm

df -h is ok:
-bash-4.1$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_firstdisk-lv_root
103G 14G 85G 14% /
tmpfs 32G 0 32G 0% /dev/shm
/dev/sda1 248M 40M 197M 17% /boot
/dev/mapper/Omero-Omero--lv1
8.8T 8.1T 800G 92% /san/omero
The filesystem is XFS. IT services have just reported they think they've fixed the problem, which was described in vague terms, something about moving things and then deleting a file and then suddenly being able to write to the SAN. I tried uploading an image using the client, it imported (but failed to create a thumbnail), then subsequent imports failed. So the problem has returned. I'll upload the logs (too big as an attachment) because I managed to find a thread where you discussed files systems and posix locking semantics, so they might help,

Bernie
bernie
 
Posts: 50
Joined: Mon Aug 17, 2009 12:03 pm

Re: large number of temporay files - is this normal

Postby jmoore » Thu Mar 20, 2014 10:40 am

Hi Bernie,

I assume the logs uploaded to https://www.openmicroscopy.org/qa2/qa2/ ... back/7968/ are yours. There are a number of these type of errors:

Code: Select all
2014-03-19 16:56:37,096 INFO  [        ome.services.util.ServiceHandler] (l.Server-6)  Excp:    ome.conditions.ResourceError: /OMERO/Pixels/Dir-092/92271 (No space left on device)

but that doesn't seem to match with your df output.

I also don't see any of the tell-tale signs of locking issues (yet).
Sorry for not being more help.
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: large number of temporay files - is this normal

Postby bernie » Thu Mar 20, 2014 10:53 am

Hi Simon, Josh, everybody,

the file system is XFS and the issue was fixed by mounting in 64bit inode addressing mode. If you Google this it's all there. This also explain the errors you found Josh.

Bernie
bernie
 
Posts: 50
Joined: Mon Aug 17, 2009 12:03 pm

Re: large number of temporay files - is this normal

Postby jmoore » Fri Mar 21, 2014 2:28 pm

Thanks for letting us know, Bernie!
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: large number of temporay files - is this normal

Postby stefanm » Fri May 02, 2014 9:42 am

Hello,

I've got a seemingly similar symptom.

Server Version: Omero-server 5.0.1 ice35 b21
OS: OpenSUSE 13.1 x64
Filesystem for the data: XFS

The server is still a deployment test install, and almost empty (around 20 image files/containers in the repository so far) and apart from importing those couple of files it's sitting there doing nothing.

However, using lsof tells me that omero has almost 2000 files open all of which are residing in .../data/FullText with the extension .cfs and all of which are marked (deleted) in lsof. The outputs of lsof and ls FullText can be found in the attachment.

Although it looks suspiciously like omero does not close (temporary) files which another process or thread has already deleted, I would be grateful if bernie could give some more details about what and why the IT department changed in the mounting options for xfs.

Best wishes
Stefan
Attachments
lsof-ls-output.zip
output of lsof (filtered for "deleted")
output of "ls -l" in ..../data/FullText/
(8.15 KiB) Downloaded 125 times
stefanm
 
Posts: 13
Joined: Fri Feb 28, 2014 3:20 pm
Location: Germany

Re: large number of temporay files - is this normal

Postby bernie » Tue May 06, 2014 8:41 am

Hi Stefan,

I've asked our ITS department to give more details over and above the explanation I've given but the person in question is away until the 9th. A similar issue arose again about 2 weeks ago which was easily cleared by rebooting. Sorry to say I have no further details about this yet,

Bernie
bernie
 
Posts: 50
Joined: Mon Aug 17, 2009 12:03 pm

Next

Return to Installation and Deployment

Who is online

Users browsing this forum: No registered users and 1 guest