We're Hiring!

OMERO Bulk import

General user discussion about using the OMERO platform to its fullest. Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

Re: OMERO Bulk import

Postby alexr » Thu Jun 28, 2018 3:19 pm

Hi Josh,
thanks for the ticket opening. I can confirm, that import with other files is working. So it seams to be specific for these files.
I have two additional questions:
Is there a way how I can get the dataset ID back after the bulk import? I mean, if I start the import using the bulk.yaml and filelist.tsv the newly added dataset is given an dataset ID by OMERO server. I then want directly to add tags (and probably key /value pairs) without looking up the dataset ID first.

The next question is related. I figured out that I can add tags by this command:
tag_ann = omero.gateway.TagAnnotationWrapper(conn)
tag_ann.setValue("New Tag")
tag_ann.save()
dataset = conn.getObject("DatasetName", ID)
dataset.linkAnnotation(tag_ann)

But can I give a list of tags / multiple tags in one go?
I tried:
tag_ann.setValue(["New Tag 1","New Tag 2","New Tag 3"])
but get the error that omero.gateway expects a string:
_textValue = <invalid value - expected ::omero::RString>

Is there a direct way to solve that or do I simply implement a loop?

Thanks
Alex
alexr
 
Posts: 46
Joined: Tue Jun 12, 2018 12:20 pm

Re: OMERO Bulk import

Postby jmoore » Fri Jun 29, 2018 10:43 am

Hey Alex,

alexr wrote:thanks for the ticket opening. I can confirm, that import with other files is working. So it seams to be specific for these files.


Good to hear that other files are working.

Is there a way how I can get the dataset ID back after the bulk import? I mean, if I start the import using the bulk.yaml and filelist.tsv the newly added dataset is given an dataset ID by OMERO server. I then want directly to add tags (and probably key /value pairs) without looking up the dataset ID first.


Sorry, no, the importer doesn't report the ID for the object that got created. (Card created). One option though would be to pre-create the dataset and then use that:

Code: Select all
DATASET=$(bin/omero -q obj new Dataset name:foo)
bin/omero import -T $DATASET my.file


Note, however, that this creates a dataset every time. To re-use a dataset, the command would need to be a bit more complex, first testing if a dataset exists.

But can I give a list of tags / multiple tags in one go?
I tried:
tag_ann.setValue(["New Tag 1","New Tag 2","New Tag 3"])
but get the error that omero.gateway expects a string:
_textValue = <invalid value - expected ::omero::RString>

Is there a direct way to solve that or do I simply implement a loop?


Here you want to create multiple tag_ann instances and link them:

Code: Select all
dataset = conn.getObject("DatasetName", ID)
for x in ("New Tag 1", "New Tag 2"):
    tag_ann = omero.gateway.TagAnnotationWrapper(conn)
    tag_ann.setValue(x)
    tag_ann.save()
    dataset.linkAnnotation(tag_ann)


cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: OMERO Bulk import

Postby alexr » Fri Jun 29, 2018 1:17 pm

Hi Josh,
thanks for the update.
when you suggest:
DATASET=$(bin/omero -q obj new Dataset name:foo)
bin/omero import -T $DATASET my.file

This only works with single file import, but in the bulk.yml import method I give the dataset name in the filelist.tsv, correct?

So to attach tags directly after import I would require to:
1st. import the files to a dataset (dataset name defined in the filelist.tsv)
2nd wait for the import to be finished
3rd get the ID for the dataset name
4th add the tags based on the dataset ID with a loop as suggested.

Now my question is for the 3rd point. How can I get the ID for a given Dataset name? And how can it be handled if multiple users have given identical dataset names, can I filter the datasets by user and then retrieve the ID?
Again I could not find an example for this problem on the omero website / git.
Thanks
Alex
alexr
 
Posts: 46
Joined: Tue Jun 12, 2018 12:20 pm

Re: OMERO Bulk import

Postby jmoore » Fri Jun 29, 2018 1:28 pm

Hey,

alexr wrote:This only works with single file import, but in the bulk.yml import method I give the dataset name in the filelist.tsv, correct?


True. You'd need to encode the ID into the tsv file.

So to attach tags directly after import I would require to:
1st. import the files to a dataset (dataset name defined in the filelist.tsv)
2nd wait for the import to be finished
3rd get the ID for the dataset name
4th add the tags based on the dataset ID with a loop as suggested.


For a dataset created during the import, yes.

Now my question is for the 3rd point. How can I get the ID for a given Dataset name? And how can it be handled if multiple users have given identical dataset names, can I filter the datasets by user and then retrieve the ID?


It's definitely possible to query for a dataset by name and user:

Code: Select all
$ bin/omero -q hql "select d.id from Dataset d where d.details.owner.id = 0 and d.name = 'josh' " --style=plain | cut -f2 -d,
24651


but you aren't sure that this is where your images are. It might be better to ask for the dataset that has one or all of your images:

Code: Select all
$ bin/omero -q hql "select d.id from Dataset d join d.imageLinks dil where dil.child.id = 132202"
# | Col1
---+-------
0 | 24651
(1 row)


Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: OMERO Bulk import

Postby alexr » Fri Jun 29, 2018 1:55 pm

Thnaks that helped.
Alex
alexr
 
Posts: 46
Joined: Tue Jun 12, 2018 12:20 pm

Re: OMERO Bulk import

Postby alexr » Tue May 07, 2019 11:41 am

Dear forum members,
after I had my inplace bulk importer working for some time I face a new problem after moving to a freshly installed Omero server with dedicated hardware.
I have again the image files on a file server that is permanently mounted to the Omero server.

The smb share is mounted in /mnt/CCHL-User/ and I checked to have read and write access for both the user that runs the omero server (user:omero) as well as the inplace user (user:inplace).
Also these both users have full access to the managed repository /OMERO/ManagedRepository/...

But when I try to run a bulk import using the two files:

bulki-59892.yml
Code: Select all
---
continue: "true"
transfer: "ln_s"
checksum_algorithm: "File-Size-64"
logprefix: "logs/"
output: "yaml"
path: "/OMERO/ManagedRepository/ipimp-59892.tsv"
columns:
   - target
   - path

ipimp-59892.tsv
Code: Select all
Dataset:name:microtus   /mnt/CCHL-User/Alex/03-Microscopy/2019/2019-02-14-microtus-meta-exp3/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi
Dataset:name:microtus   /mnt/CCHL-User/Alex/03-Microscopy/2019/2019-02-14-microtus-meta-exp3/female_m-H3K27me3-A488(2)_r-H3K9me3-A59417.zvi


using the following commands:
Code: Select all
/home/omero/OMERO.server/bin/omero --sudo inplace -w ##### -u "####" login -s 130.###.###.###

followed by
Code: Select all
/home/omero/OMERO.server/bin/omero import --bulk /OMERO/ManagedRepository/bulki-59892.yml --skip upgrade

I get the message:
Code: Select all
2019-05-07 13:25:05,939 391        [      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.4.10-ice36-b105
2019-05-07 13:25:05,962 414        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 5.9.2 revision: 9fc607f85b8900be786813296f1eee75cc1ed883 date: 31 August 2018
2019-05-07 13:25:06,038 490        [      main] INFO   formats.importer.cli.CommandLineImporter - Disabling upgrade check
2019-05-07 13:25:06,039 491        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2019-05-07 13:25:06,045 497        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2019-05-07 13:25:06,045 497        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting output format: yaml
2019-05-07 13:25:06,046 498        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2019-05-07 13:25:06,651 1103       [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2019-05-07 13:25:08,257 2709       [      main] INFO      ome.formats.importer.ImportCandidates - 1 file(s) parsed into 1 group(s) with 1 call(s) to setId in 1593ms. (1606ms total) [0 unknowns]
2019-05-07 13:25:08,342 2794       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to 130.###.###.###:4064
2019-05-07 13:25:09,276 3728       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2019-05-07 13:25:09,780 4232       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.4.10
2019-05-07 13:25:09,780 4232       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.4.10-ice36-b105
2019-05-07 13:25:09,780 4232       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 1.8.0_212
2019-05-07 13:25:09,780 4232       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2019-05-07 13:25:09,780 4232       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2019-05-07 13:25:09,781 4233       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 4.9.0-8-amd64
2019-05-07 13:25:10,477 4929       [      main] INFO          ome.formats.importer.ImportConfig - Using import target: Dataset:name:microtus
2019-05-07 13:25:10,528 4980       [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2019-05-07 13:25:11,541 5993       [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2019-05-07 13:25:11,560 6012       [3-thread-1] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /mnt/CCHL-User/Alex/03-Microscopy/2019/2019-02-14-microtus-meta-exp3/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi...
2019-05-07 13:25:11,678 6130       [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /mnt/CCHL-User/Alex/03-Microscopy/2019/2019-02-14-microtus-meta-exp3/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi
2019-05-07 13:25:11,835 6287       [3-thread-1] WARN   formats.importer.transfers.TransferState - Exception closing b534e6fe-bd64-446d-b239-523a6abfde71/a1f9e735-6c21-45f4-b0f8-c2a0cf2e4b60-RepoRawFileStoreI -t -e 1.0:tcp -h 130.83.98.66 -p 39395 -t 60000
omero.ResourceError: null
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_212]
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_212]
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_212]
   at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_212]
   at java.lang.Class.newInstance(Class.java:442) ~[na:1.8.0_212]
   at IceInternal.BasicStream.createUserException(BasicStream.java:2779) ~[ice.jar:na]
   at IceInternal.BasicStream.access$300(BasicStream.java:14) ~[ice.jar:na]
   at IceInternal.BasicStream$EncapsDecoder10.throwException(BasicStream.java:3298) ~[ice.jar:na]
   at IceInternal.BasicStream.throwException(BasicStream.java:2291) ~[ice.jar:na]
   at IceInternal.OutgoingAsync.throwUserException(OutgoingAsync.java:399) ~[ice.jar:na]
   at omero.api.RawFileStorePrxHelper.end_close(RawFileStorePrxHelper.java:1846) ~[blitz.jar:na]
   at omero.api.RawFileStorePrxHelper.close(RawFileStorePrxHelper.java:1733) ~[blitz.jar:na]
   at omero.api.RawFileStorePrxHelper.close(RawFileStorePrxHelper.java:1720) ~[blitz.jar:na]
   at ome.formats.importer.transfers.TransferState.closeUploader(TransferState.java:252) ~[blitz.jar:na]
   at ome.formats.importer.transfers.AbstractExecFileTransfer.transfer(AbstractExecFileTransfer.java:72) [blitz.jar:na]
   at ome.formats.importer.ImportLibrary.uploadFile(ImportLibrary.java:483) [blitz.jar:na]
   at ome.formats.importer.ImportLibrary$3.call(ImportLibrary.java:584) [blitz.jar:na]
   at ome.formats.importer.ImportLibrary$3.call(ImportLibrary.java:581) [blitz.jar:na]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_212]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212]
   at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
2019-05-07 13:25:11,837 6289       [3-thread-1] ERROR     ome.formats.importer.cli.ErrorHandler - FILE_EXCEPTION: /mnt/CCHL-User/Alex/03-Microscopy/2019/2019-02-14-microtus-meta-exp3/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi
java.lang.RuntimeException: omero.ResourceError : /OMERO/ManagedRepository/AlexR_2/2019-05/07/13-25-10.671/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi (Permission denied)
The server could not check the size of the file:
-----------------------------------------------
/OMERO/ManagedRepository/AlexR_2/2019-05/07/13-25-10.671/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi
-----------------------------------------------
Most likely the server process has no read access
and therefore in-place import cannot proceed. You
should delete this file manually if you are sure
that the original is safe.

   at ome.formats.importer.transfers.AbstractExecFileTransfer.checkTarget(AbstractExecFileTransfer.java:231) ~[blitz.jar:na]
   at ome.formats.importer.transfers.AbstractExecFileTransfer.transfer(AbstractExecFileTransfer.java:66) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary.uploadFile(ImportLibrary.java:483) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary$3.call(ImportLibrary.java:584) [blitz.jar:na]
   at ome.formats.importer.ImportLibrary$3.call(ImportLibrary.java:581) [blitz.jar:na]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_212]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212]
   at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
2019-05-07 13:25:11,840 6292       [2-thread-1] ERROR        ome.formats.importer.ImportLibrary - Error on import
java.lang.RuntimeException: omero.ResourceError : /OMERO/ManagedRepository/AlexR_2/2019-05/07/13-25-10.671/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi (Permission denied)
The server could not check the size of the file:
-----------------------------------------------
/OMERO/ManagedRepository/AlexR_2/2019-05/07/13-25-10.671/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi
-----------------------------------------------
Most likely the server process has no read access
and therefore in-place import cannot proceed. You
should delete this file manually if you are sure
that the original is safe.

   at ome.formats.importer.transfers.AbstractExecFileTransfer.checkTarget(AbstractExecFileTransfer.java:231) ~[blitz.jar:na]
   at ome.formats.importer.transfers.AbstractExecFileTransfer.transfer(AbstractExecFileTransfer.java:66) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary.uploadFile(ImportLibrary.java:483) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary$3.call(ImportLibrary.java:584) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary$3.call(ImportLibrary.java:581) ~[blitz.jar:na]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_212]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212]
   at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
2019-05-07 13:25:11,842 6294       [2-thread-1] INFO         ome.formats.importer.ImportLibrary - Continuing after error

==> Summary
0 files uploaded, 0 filesets created, 0 images imported, 1 error in 0:00:01.907
1 failed imports

and the import fails. I see the softlinks generated in the respective folder, but no images are imported in Omero.
I think there is a problem with file access, but I can not figure out what. The error:
omero.ResourceError : /OMERO/ManagedRepository/AlexR_2/2019-05/07/13-25-10.671/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi (Permission denied)
indicates that the server has no permission on the ManagedRepository, or does this refer to the mounted share?
I used ACL to set permissions in the managed repositories and the files I get after import look like this:
Code: Select all
-rw-rw-rw-+  1 omero inplace  27K May  7 13:27 13-27-48.191.log
drwxrwsrwx+  2 omero inplace 4.0K May  7 13:27 13-27-48.191
-rw-rw-rw-+  1 omero inplace  27K May  7 13:27 13-27-47.110.log
drwxrwsrwx+  2 omero inplace 4.0K May  7 13:27 13-27-47.110

And the link file in the directories looks like this
Code: Select all
lrwxrwxrwx   1 inplace inplace  115 May  7 13:25 female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi -> /mnt/CCHL-User/Alex/03-Microscopy/2019/2019-02-14-microtus-meta-exp3/female_m-H3K27me3-A488(2)_r-H3K9me3-A59416.zvi


Which user is in my case the one the requires access if I run the above lines for the import? Is it the inplace or the omero user?

I also checked the server status with
/bin admin diagnostics and get:
OMERO SSL port:4064
OMERO TCP port:4063
OMERO data dir:'/OMERO' Exists? True Is writable? True
OMERO temp dir:'/home/inplace/omero/tmp' Exists? True Is writable? True (Size: 0)

Thanks for the help,
Alex
alexr
 
Posts: 46
Joined: Tue Jun 12, 2018 12:20 pm

Re: OMERO Bulk import

Postby manics » Tue May 07, 2019 12:39 pm

This could be a simple permissions problem, try sudo-ing to the omero server OS user and checking you can access the symlinked file on the samba share.

However it also sounds very similar to viewtopic.php?f=5&t=8450&p=19691#p19691 If this is the problem then unfortunately there's no definitive solution. Did anything change on the samba server when you switched to the new OMERO server?
User avatar
manics
Team Member
 
Posts: 261
Joined: Mon Oct 08, 2012 11:01 am
Location: Dundee

Re: OMERO Bulk import

Postby alexr » Tue May 07, 2019 12:54 pm

Dear mancis,
thanks for the replay. I checked with both
omero and inplace user.
They can access, read and write the smb share. But you were correct. The smb share was also updated. I am currently checking the smb versions and log files to see if there is anything similar to the link you posted.
Thanks
Alex
alexr
 
Posts: 46
Joined: Tue Jun 12, 2018 12:20 pm

Re: OMERO Bulk import

Postby alexr » Tue May 07, 2019 3:37 pm

Dear manics,
I can support the problem described in this previous thread:
viewtopic.php?f=5&t=8450
After I changed my SMB server back to support smb version 1 and changed the etc/fstab on the omero server to mount the share with option vers=1.0 the inplace import is working again.
Thanks for your help. Best wishes
Alex
alexr
 
Posts: 46
Joined: Tue Jun 12, 2018 12:20 pm

Re: OMERO Bulk import

Postby manics » Tue May 07, 2019 5:09 pm

Thanks, it's useful to know it's specific to the Samba protocol version.
User avatar
manics
Team Member
 
Posts: 261
Joined: Mon Oct 08, 2012 11:01 am
Location: Dundee

Previous

Return to User Discussion

Who is online

Users browsing this forum: No registered users and 1 guest