In-place importing as sysadmin

[ratamero]

What I mean is that, since I'll be the only person running in-place imports and since those will be outside normal working hours, I can have repo path by default pointing to a subdirectory that is a share, and when in-place imports are necessary:

1) change repo path to exclude the subdirectory - new files will be saved locally on the OMERO.server machine
2) Run in-place imports - those should work since symlinks would be between the OMERO.server machine and the storage server where the data is
3) change repo path back to including the subdirectory - new files are again saved on a remote share

Is that a bit clearer? As an example, out current data dir is /data/omerodata, which is itself a symlink to a mount point. If I make that a local directory and create a new symlink for the same mount point at /data/omerodata/remote, I can include "/remote/" on my default repo path, remove it any time I need to run an in-place import, and add it again at the end of it. In my mind, that works just fine!

[jmoore]

In my mind, that works just fine!

Glad to hear it! For us, this is a new one, so we'll look forward to any feedback you have. Two other contingencies I can think of:

1. Depending on how much data you expect to see per day/month and how large your local volume can be, the default template path uses a `user/year-month/day` strategy. At the end of each month/day, you could move the directory off to larger storage and symlink assuming no imports are running [and] the directory itself doesn't contain symlinks.

2. You could use a dedicated user or group for the imports and make it's directory a symlink to local storage, and then after the import is done use `chown` or `chgrp` to "give" the data to the user. Primary downtime is the wait-time before the data appears for the user.

All the best,
~Josh

[ratamero]

OK, I finally had some time to give this a try. I started by trying the "create a dedicated user for importing and make its directory a symlink to local storage". It turns out the problem is that our high capacity storage doesn't support symlinks at all. Symlinking to local storage was impossible.

Then, I tried the convoluted idea I had. What I did was:

- make my data directory a local storage folder
- extend it
- make the subdirectory a symlink to remote storage
- run in-place import
- change repo path to include the subdirectory and point new imports to remote storage

There is one big problem, and one small problem. The small problem is that I'd have to directly manipulate the database to point to existing files in the remote storage, since they've moved to a subdirectory. I've tried "update originalfile set repo = '<HASH>' where repo = '<OTHERHASH>'" on psql and got the following error:

ERROR: cannot set original file repo property without secret key

So apparently I'm not allowed to do that. As I said, this is not a big deal; our production server is not live yet, so I could run the in-place imports before any data exists and bypass it altogether. Now comes the big problem: I can't get the in-place importing to work either. No matter which directory I try to import, with different formats and so on, I always get something like

ERROR ome.formats.importer.ImportLibrary - Error on import: cannot flush /data/omerodata/ManagedRepository/103/msrlbl_202/2018-02/21/10-18-39.911/160926_ChTOGmut_Pericentrin_IF.mvd2: java.io.IOException: Permission denied

Here, /data/omerodata is a directory in local storage. Ideas would be appreciated!

[jmoore]

ERROR: cannot set original file repo property without secret key

So apparently I'm not allowed to do that. As I said, this is not a big deal; our production server is not live yet, so I could run the in-place imports before any data exists and bypass it altogether.

Correct. Changes like this can be quite far-reaching. That being said, someone *was* doing something similarly quite recently. I'll see if I can find the thread.

Now comes the big problem: I can't get the in-place importing to work either. No matter which directory I try to import, with different formats and so on, I always get something like

ERROR ome.formats.importer.ImportLibrary - Error on import: cannot flush /data/omerodata/ManagedRepository/103/msrlbl_202/2018-02/21/10-18-39.911/160926_ChTOGmut_Pericentrin_IF.mvd2: java.io.IOException: Permission denied

Here, /data/omerodata is a directory in local storage. Ideas would be appreciated!

Urgh. Ok. Have you tried taking Java out of the picture? i.e. as the user running OMERO.server, can you also create that file? Is there any other information in the log?

Cheers,
~Josh

[jmoore]

cF https://www.openmicroscopy.org/community/viewtopic.php?f=6&t=8312

Unfortunately, looks like there wasn't an import-time solution there.

~J.

[ratamero]

First of all, sorry about the delayed answer - a bunch of stuff came up and the priority for getting this to work went way down since then.

Urgh. Ok. Have you tried taking Java out of the picture? i.e. as the user running OMERO.server, can you also create that file? Is there any other information in the log?

Cheers,
~Josh

The file (i.e. the symlink) exists, has 777 permission and is owned by the user running OMERO.server. I've tried creating the file in that folder as well, and that also works fine. The log doesn't have anything other than that Java IO Exception. I'm adding the whole output from the import command in case you can spot anything I might be missing here!

Code: Select all

Using session 6dc01c0e-1d1b-48f2-b17b-7714ee5079ef (u1473606@localhost:4064). Idle timeout: 10 min. Current group: fictitiousgroup
2018-03-12 10:39:16,158 780        [      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.4.4-ice36-b82
2018-03-12 10:39:16,181 803        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 5.7.3 revision: 34c7ca439a617601ce1ac5dcbdcdcfc6f0bfdaf1 date: 11 January 2018
2018-03-12 10:39:16,296 918        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2018-03-12 10:39:16,332 954        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2018-03-12 10:39:17,006 1628       [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2018-03-12 10:39:19,654 4276       [      main] INFO      ome.formats.importer.ImportCandidates - 24 file(s) parsed into 12 group(s) with 12 call(s) to setId in 2397ms. (2648ms total) [0 unknowns]
2018-03-12 10:39:20,379 5001       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2018-03-12 10:39:21,380 6002       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2018-03-12 10:39:22,435 7057       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.4.4
2018-03-12 10:39:22,435 7057       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.4.4-ice36-b82
2018-03-12 10:39:22,435 7057       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 1.8.0_161
2018-03-12 10:39:22,435 7057       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2018-03-12 10:39:22,435 7057       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2018-03-12 10:39:22,435 7057       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.10.0-693.21.1.el7.x86_64
2018-03-12 10:39:22,970 7592       [      main] INFO          ome.formats.importer.ImportConfig - Using import target: regex:+name:^.*u1473606/(?<Container1>.*?)
2018-03-12 10:39:23,142 7764       [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2018-03-12 10:39:24,798 9420       [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2018-03-12 10:39:24,858 9480       [      main] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /data/omeroimport/u1473606/RPE1Sec61gfp_16_11_17/RPE1-ser61bGFP_siRDNA_noco_wash1H_2_4_R3D.dv...
2018-03-12 10:39:25,030 9652       [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /data/omeroimport/u1473606/RPE1Sec61gfp_16_11_17/RPE1-ser61bGFP_siRDNA_noco_wash1H_2_4_R3D.dv
2018-03-12 10:40:45,980 90602      [      main] ERROR     ome.formats.importer.cli.ErrorHandler - FILE_EXCEPTION: /data/omeroimport/u1473606/RPE1Sec61gfp_16_11_17/RPE1-ser61bGFP_siRDNA_noco_wash1H_2_4_R3D.dv
omero.ResourceError: null
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_161]
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_161]
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_161]
   at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_161]
   at java.lang.Class.newInstance(Class.java:442) ~[na:1.8.0_161]
   at IceInternal.BasicStream.createUserException(BasicStream.java:2779) ~[ice.jar:na]
   at IceInternal.BasicStream.access$300(BasicStream.java:14) ~[ice.jar:na]
   at IceInternal.BasicStream$EncapsDecoder10.throwException(BasicStream.java:3298) ~[ice.jar:na]
   at IceInternal.BasicStream.throwException(BasicStream.java:2291) ~[ice.jar:na]
   at IceInternal.OutgoingAsync.throwUserException(OutgoingAsync.java:399) ~[ice.jar:na]
   at omero.api.RawFileStorePrxHelper.end_save(RawFileStorePrxHelper.java:724) ~[blitz.jar:na]
   at omero.api.RawFileStorePrxHelper.save(RawFileStorePrxHelper.java:611) ~[blitz.jar:na]
   at omero.api.RawFileStorePrxHelper.save(RawFileStorePrxHelper.java:598) ~[blitz.jar:na]
   at ome.formats.importer.transfers.TransferState.save(TransferState.java:128) ~[blitz.jar:na]
   at ome.formats.importer.transfers.AbstractFileTransfer.finish(AbstractFileTransfer.java:114) ~[blitz.jar:na]
   at ome.formats.importer.transfers.AbstractExecFileTransfer.transfer(AbstractExecFileTransfer.java:70) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary.uploadFile(ImportLibrary.java:437) [blitz.jar:na]
   at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:518) [blitz.jar:na]
   at ome.formats.importer.ImportLibrary.importCandidates(ImportLibrary.java:289) [blitz.jar:na]
   at ome.formats.importer.cli.CommandLineImporter.start(CommandLineImporter.java:289) [blitz.jar:na]
   at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:967) [blitz.jar:na]
2018-03-12 10:40:46,029 90651      [      main] ERROR        ome.formats.importer.ImportLibrary - Error on import: cannot flush /data/omerodata/ManagedRepository/103/u1473606_52/2018-03/12/10-39-23.457/RPE1-ser61bGFP_siRDNA_noco_wash1H_2_4_R3D.dv: java.io.IOException: Permission denied
omero.ResourceError: null
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_161]
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_161]
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_161]
   at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_161]
   at java.lang.Class.newInstance(Class.java:442) ~[na:1.8.0_161]
   at IceInternal.BasicStream.createUserException(BasicStream.java:2779) ~[ice.jar:na]
   at IceInternal.BasicStream.access$300(BasicStream.java:14) ~[ice.jar:na]
   at IceInternal.BasicStream$EncapsDecoder10.throwException(BasicStream.java:3298) ~[ice.jar:na]
   at IceInternal.BasicStream.throwException(BasicStream.java:2291) ~[ice.jar:na]
   at IceInternal.OutgoingAsync.throwUserException(OutgoingAsync.java:399) ~[ice.jar:na]
   at omero.api.RawFileStorePrxHelper.end_save(RawFileStorePrxHelper.java:724) ~[blitz.jar:na]
   at omero.api.RawFileStorePrxHelper.save(RawFileStorePrxHelper.java:611) ~[blitz.jar:na]
   at omero.api.RawFileStorePrxHelper.save(RawFileStorePrxHelper.java:598) ~[blitz.jar:na]
   at ome.formats.importer.transfers.TransferState.save(TransferState.java:128) ~[blitz.jar:na]
   at ome.formats.importer.transfers.AbstractFileTransfer.finish(AbstractFileTransfer.java:114) ~[blitz.jar:na]
   at ome.formats.importer.transfers.AbstractExecFileTransfer.transfer(AbstractExecFileTransfer.java:70) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary.uploadFile(ImportLibrary.java:437) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:518) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary.importCandidates(ImportLibrary.java:289) ~[blitz.jar:na]
   at ome.formats.importer.cli.CommandLineImporter.start(CommandLineImporter.java:289) [blitz.jar:na]
   at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:967) [blitz.jar:na]
2018-03-12 10:40:46,029 90651      [      main] INFO         ome.formats.importer.ImportLibrary - Exiting on error


[jmoore]

The file (i.e. the symlink) exists, has 777 permission and is owned by the user running OMERO.server. I've tried creating the file in that folder as well, and that also works fine.

Drat. Does your system have selinux activated? If so /var/log/audit/audit.log may have more information for us.

The log doesn't have anything other than that Java IO Exception. I'm adding the whole output from the import command in case you can spot anything I might be missing here!

The log doesn't give me any further clues, unfortunately.

~J.

[ratamero]

The system is running on a virtual machine provided by IT, which means I don't have full access (I don't have access to /var/log/audit, for example). I'll contact them and see whether they have any ideas. Thanks again!

[ratamero]

Update on this: after a long investigation, I heard back from IT services. Their message:

"Having taken a look at the 'permission denied' issue you've been having when trying to do in place imports, it looks like it is being caused by an incompatibility between Omero and SMB versions greater than 1.

By default, our provided VMs use SMB protocol version 3 when mounting to external SMB shares, the reason for this is that SMB1 is considered insecure and support for this has been deprecated by the Storage Team.

I have not had the chance to narrow down the exact file operation that Omero it trying to carry out when doing in place imports using soft symbolic links, but it looks to be related to a flush operation.

I set up a test install of Omero and a test Samba server so I could enable debug logging. When trying an import with the remote share mounted using SMB3, the following error gets logged on the Samba server

[2018/06/15 08:58:45.431921, 10, pid=7573, effective(1003, 1003), real(1003, 0)] ../source3/smbd/smb2_server.c:2279(smbd_smb2_request_dispatch)
smbd_smb2_request_dispatch: opcode[SMB2_OP_FLUSH] mid = 186821[2018/06/15 08:58:45.431969, 10, pid=7573, effective(1003, 1003), real(1003, 0)] ../source3/smbd/smb2_flush.c:134(smbd_smb2_flush_send)
smbd_smb2_flush: u1473606/RPE1Sec61gfp_16_11_17/RPE1-ser61bGFP_siRDNA_noco_wash1H_2_4_R3D.dv - fnum 2286971918
[2018/06/15 08:58:45.431998, 3, pid=7573, effective(1003, 1003), real(1003, 0)] ../source3/smbd/smb2_server.c:3120(smbd_smb2_request_error_ex)
smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1] status[NT_STATUS_ACCESS_DENIED] || at ../source3/smbd/smb2_flush.c:78

When changing the protocol to use SMB1, this does not occur and the import succeeds, without issue. I have also tried this using a Windows server to provide the share, to rule out any issues with Samba, and the same behaviour occurs."

They've also sent me the import log and smbd log (https://pastebin.com/TQTg0nHv and https://pastebin.com/xTfVi6pC)


It seems like using SMB1 version is the only thing that made the symlink problem go away, but we are (of course) a bit wary about using SMB1. Is in-place importing from an SMB3 share a known issue?

[mtbc]

Do we have a Blitz-0.log from the server with the failing import? The exception stacktrace from that could help us to pin down exactly which operation via the Java API is triggering the problem.

Wholly appreciate your aversion to SMB1. I don't believe we're aware of any other OMERO sites reporting this kind of Samba issue: I hope others may read and comment. For example, I think the groups at Oxford used to have their OMERO see the data via a Samba mount. It's possible we need to expand our local testing infrastructure to cover this case.

Incidentally, regarding your problem above with "cannot set original file repo property without secret key" there is a nasty hack that this may help illustrate where the filename gets momentarily prefixed by the node UUID:

Code: Select all

omero=> SELECT uuid FROM node WHERE down IS NULL;
                 uuid                 
--------------------------------------
33534e55-bccc-4784-b65f-7de1ea1b8c06
(1 row)

omero=> SELECT repo, name FROM originalfile WHERE id = 102;
                 repo                 |       name       
--------------------------------------+------------------
d2f6c2b8-a028-4e66-ac77-0e145e3f5e2e | 12-50-13.497.log
(1 row)

omero=> UPDATE originalfile SET name = '33534e55-bccc-4784-b65f-7de1ea1b8c0612-50-13.497.log', repo = 'foo' WHERE id = 102;
UPDATE 1
omero=> SELECT repo, name FROM originalfile WHERE id = 102;
repo |       name       
------+------------------
foo  | 12-50-13.497.log
(1 row)


Cheers,
Mark