We're Hiring!

delete files in binary repository (omero.insight crashed)

General user discussion about using the OMERO platform to its fullest. Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

delete files in binary repository (omero.insight crashed)

Postby klh » Fri Apr 20, 2018 6:08 pm

Hi OMERO team,
I'm new to OMERO, and I need your advice on how to proceed to delete files from OMERO's binary repository when the OMERO.insight client crashed, probably after the upload was completed(?). When I login into omero.web client, I do not see any data files shown for that user that uploaded. For details, please read below.

I recently installed OMERO.server 5.4.5 (on a RHEL 7.4). I also installed the omero.web client (installed separately from the OMERO.server). I have used OMERO.insight-5.4.5 client to upload some tiff files (<50) to our OMERO.server and everything works great.

Yesterday I started a upload via OMERO.insight: I selected 720 tiff files (less than 2GB) from one of our OperaPhenix HCS data sets (full data set is ~86GB, has 55440 tiff files+1 Index.idx.xml file) after a while I noticed that OMERO.insight is actually trying to upload the entire data set. I suspect it tries to uploads every file listed in the Index.idx.xml file (is that correct?) I didn't interrupt, and let it go overnight.This morning OMERO.insight showed it has uploaded 86GB, see attached screenshot. But also OMERO.insight client seemed to have crashed at some point (was not responding etc). I looked at the OMERO.insight log file and it shows:
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "XToolkt-Shutdown-Thread"
Exception in thread "TimerQueue" java.lang.OutOfMemoryError: Java heap space

When I login into omero.web client, I do not see any data files shown for that user that uploaded, I see only one single "screen" folder that is empty, I looked under OMERO's binary repository, I see this directory:
/data/omero/omerotest/ManagedRepository/operaphenix_102/2018-04/19/10-32-32.996/All_Images/

Doing "du -sh All_Images/" shows: 109G and "ls -l All_Images/| wc -l" shows: 55442
So it seems all the tiff files were uploaded but it never made it into OMERO database?
I read the directions given here: https://docs.openmicroscopy.org/omero/5.4.5/developers/Modules/Delete.html and here: https://www.openmicroscopy.org/community/viewtopic.php?f=4&t=8193&p=17851&hilit=delete+files+in+binary+repository#p17851

Is there anything else I need to watch out for when trying to delete these 86GB of files that do not show up in omero.web ? any tips, suggestions are much appreciated.

Kia
Attachments
OMERO-insight_uploaded_86GB.png
OMERO-insight_uploaded_86GB.png (50.53 KiB) Viewed 1332 times
klh
 
Posts: 4
Joined: Fri Oct 13, 2017 2:25 pm

Re: delete files in binary repository (omero.insight crashed

Postby mtbc » Mon Apr 23, 2018 9:12 am

Dear Kia,

Insight certainly has limits on what it can import. You can increase OMERO.insight's memory using http://help.openmicroscopy.org/configur ... l#increase but for large-scale imports OMERO.cli at http://docs.openmicroscopy.org/latest/o ... mport.html is often superior.

To clean up the files it is probably best to delete the failed fileset. It's great that you found their path as you can find the fileset ID from the command line using something like,

Code: Select all
bin/omero hql --all "SELECT DISTINCT fileset.id FROM FilesetEntry WHERE originalFile.path LIKE 'operaphenix_102/2018-04/19/10-32-32.996/%'"

To double-check, given a high enough limit to reach far enough back in time, commands like,

Code: Select all
bin/omero fs sets --order newest --limit 20

should show that failed fileset having many files but no images, with the same ID. It can be deleted with,

Code: Select all
bin/omero delete Fileset:12345

where the "12345" is the fileset ID shown from the above commands. You can then try the above commands again to check that the fileset is gone. The files should now be gone from disk too.

Also note the end of http://docs.openmicroscopy.org/latest/o ... inary-data about how "bin/omero admin cleanse" will delete any empty directories left behind.

If you have any questions about the above then please do not hesitate to ask further. This kind of thing is well worth getting right.

Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Re: delete files in binary repository (omero.insight crashed

Postby mtbc » Wed Apr 25, 2018 7:35 am

A postscript for anybody coming back to this thread, finding it via search or similar: I now notice that "bin/omero fs sets" offers a "--without-images" option that will pick out these failed imports.
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland


Return to User Discussion

Who is online

Users browsing this forum: No registered users and 1 guest