This documentation is for the new OMERO 5.3 version. See the latest OMERO 5.2.x version or the previous versions page to find documentation for the OMERO version you are using if you have not upgraded yet.
In-place import is a feature added to OMERO 5.0.0 to allow files which are already present on the server machine to be imported into OMERO without the need to copy them. This requires users to have shell (SSH, etc.) access to the server machine, and so there are a number of limitations to this implementation. Development of this feature is on-going, with improvements planned to enable a more user-friendly experience. This CLI-based stop-gap is being made available at this stage because for some users, in-place import is essential for their use of OMERO.
This feature is designed to allow imaging facilities to import large datasets into OMERO while keeping them safely stored in a secure repository which is read-only for users. Leaving the data in a user’s file system is very dangerous as they may forget they need to keep it or move to a different institution. Under no circumstances should in-place import be used with temporary storage.
The instructions below should help you get started but it is critical that you understand the implications of using this feature. Please do not just copy commands and hope for the best.
As a data management platform, OMERO assumes that it is in control of your data in order to help prevent data loss. It assumes that data was copied into the server and only a server administrator or authorized OMERO user would have the rights to do anything destructive to that data.
With in-place import, the data either resides completely outside of OMERO or is shared with other users. This means that the critical, possibly sole, copy of your data must be protected outside of OMERO. This is your responsibility for the lifetime of the data.
In-place import is only available on the OMERO server system itself. In other words, using SSH or similar, you will need to shell into the server and run the command-line importer directly. If you are uncomfortable with this, you should let someone else handle in-place importing.
Someone wanting to perform an in-place import MUST have:
The above means that it may be useful to create a single OS account (e.g. “import_user”) which multiple users can log into, and then use their own OMERO accounts to import data. Alternatively, each OMERO user can be given an OS account with access rights to the data storage as well as the managed repository.
At the moment full in-place import functionality is available only under POSIX systems (Unix, Linux, OS X). Under Windows, ln and ln_s work as expected but ln_rm does not automatically remove image files after import.
For soft linking with --transfer=ln_s it has been noticed that some plate imports run rather more slowly than usual. Other operations may also be affected. In determining if or how to use in-place import at your high-content screening facility, we thus recommend time profiling with representative data, and alerting us to any significant disappointments.
Also, there is still some data duplication when pyramids are generated. We are hoping to find a work-around for this in the future.
Whether you chose to use the hard- or soft-linking option below, you should take steps to secure files which are in-place imported to OMERO. The best option is making them read-only for both the OMERO user and also for the owner of the data. This means the server cannot accidentally modify the files (e.g. if a client mixes up the file IDs and tries to write to the wrong location) and that the files cannot be removed from the system while OMERO is still using them. Files may not be renamed or otherwise altered such that the OMERO server user cannot find them at the expected location.
If possible, all the files should be added to your regular backup process. If the files for imported images are later removed or corrupted, the result will probably be that while the images remain in their projects or screens with their annotations and basic metadata, they simply cannot be successfully viewed. However, this behavior is not guaranteed, so do not assume that the resulting problems will not extend further. Once the problem is noticed, replacing the original image files from backups, in the same place with the same name, is likely but not guaranteed to fully restore the images and their associated data in OMERO.
In-place import requires additional user and group setup. As no-one should be allowed to log into the account used to install the server, to permit in-place imports you need to create a different user account, allowing someone to log into the server but not accidentally delete any files. Therefore, you should set up an ‘in-place’ user and an ‘in-place’ group and configure a subset of directories under ManagedRepository to let members of that group write to them. The example below details how this was done for one of our test servers in Dundee:
### STATUS BEFORE [sysadmin@ome-server omero_system_user]$ umask 0002 [sysadmin@ome-server omero_system_user]$ ls -ltrad ManagedRepository/ drwxrwxr-x 8 omero_system_user omero_system_user 4096 Apr 24 10:13 ManagedRepository/ [sysadmin@ome-server omero_system_user]$ grep inplace /etc/passwd /etc/group /etc/passwd:inplace_user:x:501:501::/home/inplace_user:/bin/bash /etc/group:omero_system_user:x:500:inplace_user /etc/group:inplace_user:x:501: [sysadmin@ome-server omero_system_user]$ grep omero_system_user /etc/passwd /etc/group /etc/passwd:omero_system_user:x:500:500::/home/omero_system_user:/bin/bash /etc/group:omero_system_user:x:500:inplace_user [sysadmin@ome-server omero_system_user]$ sudo -u inplace_user -i [inplace_user@ome-server ~]$ umask 0002 ### SCRIPT chgrp inplace_user /repositories/binary-repository/ManagedRepository chmod g+rws /repositories/binary-repository/ManagedRepository chmod g+rws /repositories/binary-repository/ManagedRepository/* chmod g+rws /repositories/binary-repository/ManagedRepository/*/* chmod g+rws /repositories/binary-repository/ManagedRepository/*/*/* chgrp inplace_user /repositories/binary-repository/ManagedRepository/* chgrp inplace_user /repositories/binary-repository/ManagedRepository/*/* chgrp inplace_user /repositories/binary-repository/ManagedRepository/*/*/* # With the above, newly created directories should be in the inplace group # As long as the file is readable by omero_system_user, then it should work fine! ### AFTER SCRIPT [root@ome-server omero_system_user]# ls -ltrad ManagedRepository/ drwxrwsr-x 8 omero_system_user inplace_user 4096 Apr 24 10:13 ManagedRepository/ ### TEST # with default umask this likely has to do [inplace_user@ome-server ~]$ cd /repositories/binary-repository/ManagedRepository/ [inplace_user@ome-server ManagedRepository]$ mkdir inplace.test [inplace_user@ome-server ManagedRepository]$ ls -ltrad inplace.test/ drwxrwsr-x 2 inplace_user inplace_user 4096 Apr 30 11:35 inplace.test/ [omero_system_user@ome-server omero_system_user]$ cd /repositories/binary-repository/ManagedRepository/ [omero_system_user@ome-server ManagedRepository]$ rmdir inplace.test/ [omero_system_user@ome-server ManagedRepository]$
From 5.0.0, the command-line import client has a new help menu which explains the available options:
$ bin/omero import --advanced-help
ADVANCED OPTIONS: These options are not intended for general use. Make sure you have read the documentation regarding them. They may change in future releases. In-place imports: ----------------- --transfer=ARG File transfer method General options: upload # Default upload_rm # Caution! File upload followed by source deletion. some.class.Name # Use a class on the CLASSPATH. Server-side options: ln # Use hard-link. ln_s # Use soft-link. ln_rm # Caution! Hard-link followed by source deletion. cp # Use local copy command. cp_rm # Caution! Copy followed by source deletion. e.g. $ bin/omero import -- --transfer=ln_s foo.tiff $ ./importer-cli --transfer=ln bar.tiff $ CLASSPATH=mycode.jar ./importer-cli --transfer=com.example.MyTransfer baz.tiff Background imports: ------------------- --auto-close Close completed imports immediately. --minutes-wait=ARG Choose how long the importer will wait on server-side processing. ARG > 0 implies the number of minutes to wait. ARG = 0 exits immediately. Use a *-completed option to clean up. ARG < 0 waits indefinitely. This is the default. --close-completed Close completed imports. --wait-completed Wait for all background imports to complete. e.g. $ bin/omero import -- --minutes-wait=0 file1.tiff file2.tiff file3.tiff $ ./importer-cli --minutes-wait=0 some_directory/ $ ./importer-cli --wait-completed # Waits on all 3 imports. File exclusion: --------------- --exclude=filename Exclude files based on filename. --exclude=clientpath Exclude files based on the original path. e.g. $ bin/omero import -- --exclude=filename foo.tiff # First-time imports $ bin/omero import -- --exclude=filename foo.tiff # Second-time skips Import speed: ------------- --checksum-algorithm=ARG Choose a possibly faster algorithm for detecting file corruption, e.g. Adler-32 (fast), CRC-32 (fast), File-Size-64 (fast), MD5-128, Murmur3-32, Murmur3-128, SHA1-160 (slow, default) e.g. $ bin/omero import -- --checksum-algorithm=CRC-32 foo.tiff $ ./importer-cli --checksum-algorithm=Murmur3-128 bar.tiff --no-stats-info Disable calculation of minima and maxima when as part of the Bio-Formats reader metadata e.g. $ bin/omero import -- --no-stats-info foo.tiff $ ./importer-cli --no-stats-info bar.tiff --no-thumbnails Do not perform thumbnailing after import e.g. $ bin/omero import -- --no-thumbnails foo.tiff $ ./importer-cli --no-thumbnails bar.tiff --no-upgrade-check Disable upgrade check for each import e.g. $ bin/omero import -- --no-upgrade-check foo.tiff $ ./importer-cli --no-upgrade-check bar.tiff Feedback: --------- --qa-baseurl=ARG Specify the base URL for reporting feedback e.g. $ bin/omero import broken_image.tif -- --email EMAIL --report --upload --logs --qa-baseurl=https://qa.staging.openmicroscopy.org/qa $ ./importer-cli broken_image.tif --email EMAIL --report --upload --logs --qa-baseurl=https://qa.staging.openmicroscopy.org/qa Report bugs to <firstname.lastname@example.org>
In versions prior to 5.0.3 this help option is hidden and it can only be accessed using:
$ bin/omero import -- --advanced-help
The option for performing an in-place transfer is --transfer. A new extension point, file transfers allow a choice of which mechanism is used to get a file into OMERO.
$ bin/omero import -- --transfer=ln_s my_file.dv
Using session bba923bb-cf0c-4cf0-80c5-a309be523ad8 (root@localhost:4064). Idle timeout: 10.0 min. Current group: system ...[ main] INFO ome.formats.importer.ImportConfig - OMERO Version: 5.0.0-rc1-DEV-ice35 ...[ main] INFO ome.formats.importer.ImportConfig - Bioformats version: 5.0.0-rc1-DEV-ice35 revision: 101008f date: 31 January 2014 ...[ main] INFO formats.importer.cli.CommandLineImporter - Setting transfer to ln_s ...[ main] INFO formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO ...[ main] INFO ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM ...[ main] INFO ome.formats.importer.ImportCandidates - 1 file(s) parsed into 1 group(s) with 1 call(s) to setId in 98ms. (100ms total) [0 unknowns] ...[ main] INFO ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064 ...[ main] INFO ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back ...[ main] INFO ome.formats.OMEROMetadataStoreClient - Server: 5.0.0 ...[ main] INFO ome.formats.OMEROMetadataStoreClient - Client: 5.0.0-rc1-DEV-ice35 ...[ main] INFO ome.formats.OMEROMetadataStoreClient - Java Version: 1.7.0_51 ...[ main] INFO ome.formats.OMEROMetadataStoreClient - OS Name: Linux ...[ main] INFO ome.formats.OMEROMetadataStoreClient - OS Arch: amd64 ...[ main] INFO ome.formats.OMEROMetadataStoreClient - OS Version: 3.8.0-27-generic ...[ main] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION ...[ main] INFO ome.formats.importer.ImportConfig - OMERO Version: 5.0.0-rc1-DEV-ice35 ...[ main] INFO ome.formats.importer.ImportConfig - Bioformats version: 5.0.0-rc1-DEV-ice35 revision: 101008f date: 31 January 2014 ...[ main] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START ...[ main] INFO s.importer.transfers.SymlinkFileTransfer - Transferring /tmp/a.fake... ...[ main] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /tmp/a.fake ...[ main] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /tmp/a.fake ...[ main] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END ...[l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5 Logfile: 24605 ...[l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5 Logfile: 24605 ...[l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5 Logfile: 24605 ...[l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5 Logfile: 24605 ...[l.Client-1] INFO ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5 Logfile: 24605 ...[l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /tmp/a.fake Imported pixels: 5001 Other imported objects: Fileset:4102 Image:5001 ...[l.Client-0] INFO ome.formats.importer.cli.ErrorHandler - Number of errors: 0
The only visible difference here is the line:
...formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
Rather than uploading via the OMERO API, the command-line importer makes a call to the system ln command.
Previously, OMERO only offered the option of uploading via the API. Files were written in blocks via the RawFileStore interface. With in-place import, several options are provided out of the box as well as the ability to use your own.
Finally, the least favored option is ln_rm. It first performs a hard-link like ln, but once the import is complete it attempts to delete the original file. This is currently in testing as an option for DropBox but is unlikely to be of use to general users. Although this option is more limited than the upload_rm option below it will be much faster.
This option, available from OMERO 5.0.3, is not strictly an in-place option but is detailed here for convenience. It first performs a file upload like default import, but once the import is complete it attempts to delete the original files. It deletes the original files if and only if the import is successful.
New in 5.0.7, the cp and cp_rm commands provide the same functionality as ln and ln_rm but perform a copy rather than a link operation. The benefit of a copy is that it works over OS filesystem boundaries while still providing the integrity that ln_s cannot. The primary downside of a raw cp compared to ln is that there is data duplication. cp_rm being very similar to ln_rm usually works around this downside, except in the case of a failed import. Then the duplicated data will remain in OMERO and an explicit cleanup step will need to be taken.
If none of the above options work for you, it is also possible to write your own implementation of the ome.formats.importer.transfers.FileTransfer class, likely subclassing ome.formats.importer.transfers.AbstractFileTransfer or ome.formats.importer.transfers.AbstractExecFileTransfer. If you do so, please let us know how we might improve either the interface or the implementations that we provide.
Once your implementation has been compiled into a jar and placed in the lib/clients directory, you can invoke it using:
$ bin/omero import -- --transfer=example.package.ClassName ...