Page 1 of 1

Dropbox: clean up (remove) imported files

PostPosted: Wed Dec 10, 2014 1:16 pm
by ehrenfeu
Dear all,

we're having the following scenario: OMERO 5.0.5 is running on a dedicated machine with the space for the binary repository being provided by a central storage machine via NFS. To use OMERO.dropbox, our server has an additional ~1TB of local storage attached, which is configured as the "watchDir" and being exposed to the users via Samba / CIFS.

From the thread auto-importer I'm reading that we could configure OMERO to remove files from the watchDir once the import has succeeded using
Code: Select all
--transfer=upload_rm


Is this correct? From how I understand, we can't use the "ln_rm" option as our watchDir and the ManagedRepository reside on independent file systems each, right?

Thanks
~Niko

Re: Dropbox: clean up (remove) imported files

PostPosted: Wed Dec 10, 2014 8:08 pm
by jmoore
Hi Niko,

ehrenfeu wrote:From the thread auto-importer I'm reading that we could configure OMERO to remove files from the watchDir once the import has succeeded using --transfer=upload_rm


Yup, that's just what it's intended for.

From how I understand, we can't use the "ln_rm" option as our watchDir and the ManagedRepository reside on independent file systems each, right?


Also correct: regular hard-linking boundaries apply. And a "ln_s_rm" wouldn't get you very much! :) (That being said, if you'd like to try out a "cp_rm" we could certainly give it a try.)

Cheers,
~Josh.

Re: Dropbox: clean up (remove) imported files

PostPosted: Fri Dec 12, 2014 3:54 pm
by ehrenfeu
Hi Josh,

thanks for the update! Are we correct in assuming this option has to go here?

Code: Select all
<property name="omero.fs.importArgs"  value=""/>


Cheers
~Niko

Re: Dropbox: clean up (remove) imported files

PostPosted: Mon Dec 15, 2014 8:22 am
by cblackburn
Hi Niko,

Yes, that's correct. You can also set the configuration via the command line, as illustrated here,

https://www.openmicroscopy.org/site/sup ... ml#dropbox

Cheers,

Colin

Re: Dropbox: clean up (remove) imported files

PostPosted: Mon Dec 22, 2014 4:05 pm
by ehrenfeu
Thanks for the info! We'll try and see how it goes...

Josh, could you tell us details what's different in "cp_rm"?

Thanks
~Niko

Re: Dropbox: clean up (remove) imported files

PostPosted: Tue Dec 23, 2014 9:32 am
by jmoore
Hi Niko,

`cp_rm` doesn't exist yet, but it's a pretty simple manner of modifying this line in HardlinkFileTransfer to use "cp" rather than "ln". The difference is that a full copy takes place but locally and likely much faster than uploading via Ice (though I haven't compared the speed yet).

Cheers,
~Josh.

Re: Dropbox: clean up (remove) imported files

PostPosted: Mon Jan 26, 2015 10:37 am
by jmoore
Hi Niko,

just a follow-up that https://github.com/openmicroscopy/openm ... /pull/3379 has been merged so `cp_rm` should be in the nightly builds as well as 5.0.7 for your review.

All the best,
~Josh.