large number of temporay files - is this normal

[bernie]

Hi,

we've just moved OMERO to new hardware. All our data files are stored on the University SAN and the new server is pointing to these. The server has been built in close collaboration with the University IT department and is under Puppet management. Currently the SAN shows that there is 800G free out of a total of 8.8TB (92% utilisation). Nevertheless we cannot right to the SAN; we get the error: No space left on device. If you list open files though you can see that all available space is filed with temporary files help open by JAVA processes associated with OMERO (stop OMERO and they disappear) (see attachment). This wasn't a problem with the previous server so I'm wondering if this is normal? I've also attached the server diagnostics.

[manics]

Hi

You've only got 118 files open which looks normal and should be well below the OS limit. The files under FullText are related to search indexing. Could the inability to write to the free space be due to storage restrictions, e.g. some space being reserved for admin purposes only?

What else is being stored on the SAN? Can you write to it outside of OMERO (but as the OMERO user?).

Simon

[bernie]

Hey Simon,

well I think the IT dept is retreating from their original position and accepting there might be a problem with the SAN or how the new server is connecting to the SAN. Cannot write to the SAN when the services aren't running. The SAN stores the campus! but seems to be running normally otherwise. It's reassuring to know that files that are open are as expected. We also had to change the omero id on the new server and that now corresponds to the old omero id. I shall update this post when I know more,

Bernie

[manics]

Hi Bernie

A colleague just pointed out that it could be another partition that's filled up (e.g. a temp directory). What does df -h show? Presumably /OMERO is a symlink to your SAN? If you still have problems could you give us your server logs, either as an attachment here or upload them to https://www.openmicroscopy.org/qa2/qa/upload/

Cheers

Simon

[bernie]

df -h is ok:
-bash-4.1$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_firstdisk-lv_root
103G 14G 85G 14% /
tmpfs 32G 0 32G 0% /dev/shm
/dev/sda1 248M 40M 197M 17% /boot
/dev/mapper/Omero-Omero--lv1
8.8T 8.1T 800G 92% /san/omero
The filesystem is XFS. IT services have just reported they think they've fixed the problem, which was described in vague terms, something about moving things and then deleting a file and then suddenly being able to write to the SAN. I tried uploading an image using the client, it imported (but failed to create a thumbnail), then subsequent imports failed. So the problem has returned. I'll upload the logs (too big as an attachment) because I managed to find a thread where you discussed files systems and posix locking semantics, so they might help,

Bernie

[jmoore]

Hi Bernie,

I assume the logs uploaded to https://www.openmicroscopy.org/qa2/qa2/ ... back/7968/ are yours. There are a number of these type of errors:

Code: Select all

2014-03-19 16:56:37,096 INFO  [        ome.services.util.ServiceHandler] (l.Server-6)  Excp:    ome.conditions.ResourceError: /OMERO/Pixels/Dir-092/92271 (No space left on device)


but that doesn't seem to match with your df output.

I also don't see any of the tell-tale signs of locking issues (yet).
Sorry for not being more help.
~Josh.

[bernie]

Hi Simon, Josh, everybody,

the file system is XFS and the issue was fixed by mounting in 64bit inode addressing mode. If you Google this it's all there. This also explain the errors you found Josh.

Bernie

[jmoore]

Thanks for letting us know, Bernie!
~Josh.

[stefanm]

Hello,

I've got a seemingly similar symptom.

Server Version: Omero-server 5.0.1 ice35 b21
OS: OpenSUSE 13.1 x64
Filesystem for the data: XFS

The server is still a deployment test install, and almost empty (around 20 image files/containers in the repository so far) and apart from importing those couple of files it's sitting there doing nothing.

However, using lsof tells me that omero has almost 2000 files open all of which are residing in .../data/FullText with the extension .cfs and all of which are marked (deleted) in lsof. The outputs of lsof and ls FullText can be found in the attachment.

Although it looks suspiciously like omero does not close (temporary) files which another process or thread has already deleted, I would be grateful if bernie could give some more details about what and why the IT department changed in the mounting options for xfs.

Best wishes
Stefan

[bernie]

Hi Stefan,

I've asked our ITS department to give more details over and above the explanation I've given but the person in question is away until the 9th. A similar issue arose again about 2 weeks ago which was easily cleared by rebooting. Sorry to say I have no further details about this yet,

Bernie