OMERO Bulk import

[alexr]

Hi josh,
this is strange, first of all it was working in the beginning and then stopped, without changing the permissions or anything else. I was just playing with the bulk.yml file.
Then I checked the permissions:
The structure is like this:
OMERO|
ManagedRepository|
.omero|
repository|
9f6b... |
repo_uuid

for all directories starting at ManagedRepsoitories the permissions are like this:
owner: inplace (my OMERO / system inplace importer)
group: omerouser (my system account that runs the server)
and the permissions are set to 775 for all folders.
Nevertheless if I try to copy some text file as omerouser from the omeruser desktop to the repository I get a "permission denied".
If I do that as inplace it works.
inplace and omeruser are members of the same groups.
I am lost!

Alex

[alexr]

Hi Josh,
just a small update.
After I send the last error message, OMERO was not able to import anything et all. Even the insight client could not import anything directly. So I rebooted the complete computer and everything was fine again. Permissions are the same as before, but the system is working.
Best wishes
Alex

[jmoore]

Wow. Thanks for letting us know, Alex. Obviously, it's also disconcerting. If you ever figure out what happened, don't hesitate to let us know.

~Josh

[alexr]

Hi Josh,
I havent' figured out how and why that happend so far.

I am currently facing an other obstacle:
As you might remember I try to have a GUI tool for batch inplace import for my users. Therefore I have written a small python app that generates the bulk.yml and the fileList.tsv files, translates the pathes of the files to import relative to the mount points of the file server on the Omero machine and then issues the two commands for bulk import:

Code: Select all

home/omerouser/OMERO.server/bin/omero --sudo inplace -w'pw_for_inplace' -u 'UserToImportTo' login 


and then:

Code: Select all

/home/omerouser/OMERO.server/bin/omero import --bulk /OMERO/ManagedRepository/bulki.yml


with the two uploaded files: bulk.yml and toimport.tsv.

This works well if I let the python script issue all the commands (the python script is run on a client computer), then I transfer the two command lines from python to the Omero srever by ssh and here comes the problem:
after the first command (above) I am requested to hit enter again, although I have provided passwords etc in the command:

Code: Select all

Previously logged in to localhost:4064 as Alex R
Server: [localhost:4064]


Is there a way to overcome this enter typing? I also tried to send a "enter" via SSH, but this does not work.

On related question:
I tried also for the script to extract valid users and groups from the Omero using the pyhton API:

Code: Select all

import logging #required for Blitz
import inspect
import os
from omero.gateway import BlitzGateway
conn = BlitzGateway('omero','admin_password',host='130.83.98.66', port=4064)
conn.connect()
group = conn.getGroupFromContext()
owners, members = group.groupSummary()

user_list ="\"\""
for m in members:
    user_list = user_list +",\""+ m.getOmeName() + "\""


This is intended to avoid that the user enters non valid user names, by providing a drop down with registered users.
This works if I run it as python 2.7. As far as I understood the omero modules are all python 2.7 and will stay like this for some time. Since all my other stuff is in python 3.4, is there a easy fix for updating just the parts from omero.gateway, that are required here?
Also is there a way to retrieve a list of all groups in the omero server?
Maybe there is an easier way altogether?

Thanks for your help,
Alex

[jmoore]

This works well if I let the python script issue all the commands (the python script is run on a client computer), then I transfer the two command lines from python to the Omero srever by ssh and here comes the problem:
after the first command (above) I am requested to hit enter again, although I have provided passwords etc in the command:

Code: Select all

Previously logged in to localhost:4064 as Alex R
Server: [localhost:4064]


Is there a way to overcome this enter typing? I also tried to send a "enter" via SSH, but this does not work.

When you login on the client, a session is created (and logged to stderr) and a file is created under ~/.omero/sessions. It's this that is missing on the server-side. One option would be to pass `-k $UUID` on the server-side.

On related question:
...
This is intended to avoid that the user enters non valid user names, by providing a drop down with registered users.
This works if I run it as python 2.7. As far as I understood the omero modules are all python 2.7 and will stay like this for some time. Since all my other stuff is in python 3.4, is there a easy fix for updating just the parts from omero.gateway, that are required here?

Not that we know of. Suggestions welcome, but unfortunately due to the Ice-generated code, we understand it to be a substantial undertaking.

Also is there a way to retrieve a list of all groups in the omero server?

Sure.

Code: Select all

In [5]: [x.name for x in conn.listGroups()]
Out[5]:
[u'system',
u'user',
u'guest',
...


~Josh

[alexr]

Hi josh,
thanks again, the last bit worked fine.
I have some trouble understanding the first part of your answer:

When you login on the client, a session is created (and logged to stderr) and a file is created under ~/.omero/sessions. It's this that is missing on the server-side. One option would be to pass `-k $UUID` on the server-side.

First I do not see the /.omero/sessions. This file/folder does not exist. Do I look at the right place:
/OMERO/.omero/ this only has a subdirectory called: repository and within is the 9f6b12.... which is some days old.
This is maybe due to the fact that I do not log into OMERO with the insight client, but just connect by ssh and issue the two commands on the server?

Although when I run the first command:

Code: Select all

/home/omerouser/OMERO.server/bin/omero --sudo inplace -w'inplace password' -u 'Alex R' login


and hit enter I get:
Created session 538194f6-34a7-4743-af56-8fe9ffa85b0b
is this the UUID?
but if I then issue the command:

Code: Select all

/home/omerouser/OMERO.server/bin/omero import --bulk /OMERO/ManagedRepository/bulki.yml -k $UUID


I get an error:

Code: Select all

/home/omerouser/OMERO.server/bin/omero import: error: argument -k/--key: expected one argument


if I run instead

Code: Select all

/home/omerouser/OMERO.server/bin/omero import --bulk /OMERO/ManagedRepository/bulki.yml -k 538194f6-34a7-4743-af56-8fe9ffa85b0b


I have to again hit enter and then the import starts.

Thanks
Alex

[jmoore]

First I do not see the /.omero/sessions. This file/folder does not exist. Do I look at the right place:
/OMERO/.omero/ this only has a subdirectory called: repository and within is the 9f6b12.... which is some days old.

Sorry, `~` refers to the home directory of whatever user was running the command. i.e. `$HOME/omero/sessions`. (Sorry for the extra dot above!)

Created session 538194f6-34a7-4743-af56-8fe9ffa85b0b
is this the UUID?

Yes, "538194f6-34a7-4743-af56-8fe9ffa85b0b" (Be careful posting these on the forums as they allow someone to temporarily login as you!)

Code: Select all

/home/omerouser/OMERO.server/bin/omero import --bulk /OMERO/ManagedRepository/bulki.yml -k $UUID


What does "echo $UUID" show? Sounds like it has a space in it.

Code: Select all

/home/omerouser/OMERO.server/bin/omero import --bulk /OMERO/ManagedRepository/bulki.yml -k 538194f6-34a7-4743-af56-8fe9ffa85b0b


I have to again hit enter and then the import starts.

Hmmm.... does the "hit enter" problem go away if you include the server "-s host"?

~Josh

[alexr]

Hi Josh,
just to let you know, this last suggestion works.
1st I found the session info !
2nd adding the -s option to the import command solved the issue with hitting enter. The -k $UUID is not required.
With the $UUID I still have a problem, because the echo $UUID returned an empty string.

But the rest works fine.

A new question: can omero report back to the shell when the import is finished, so that the yml and tsv files can be deleted?

Best wishes
Alex

[jmoore]

With the $UUID I still have a problem, because the echo $UUID returned an empty string.

Ah, sorry. I see the confusion. I was assuming you'd set the variable UUID somewhere. If not, then simply replace my uses of '$UUID' with the original string.

A new question: can omero report back to the shell when the import is finished, so that the yml and tsv files can be deleted?

Hmm.... I'm a bit confused. When the shell command exits, there's definitely a return code that would let you know if the import was successful or not. Maybe I've lost the bigger picture.

~Josh.

[alexr]

Hi Josh,
you were right, there is an output send back.

Here again I describe the bigger picture:
I have written a small python GUI that allows the user (no CLI lovers) to select folders on the fileserver and have them inplace imported into the Omero server. Therefore the fileserver is mounted on a permanent mount point on the omero server and the user/clients connect to the fileserver to select the folders they want imported. It looks roughly like the attachment.

Screen Shot 2018-06-19 at 14.12.32_s.png (28.18 KiB) Viewed 1656 times

The routine is then:
[*] User selects a folder on the fileserver
[*] User enters a Dataset name
[*] User selects OMERO user from dropdown
[*] User presses "Generate List": this produces the filelist.tsv with path translated to the Omero-server-mount point of the fileserver also a bulki.yaml file is gnerated (although they are at the moment always identical) [*] These files are temporary saved localy
[*] User presses "Import to Omero": This does the following:
[*] a sftp connection is opend from the client/user machine to the Omero server and the two files (bulki.yaml and filelist.tsv) are transfered to the folder OMERO/ManagedRepositories
[*] via SSH the first command is issued:
/home/omerouser/OMERO.server/bin/omero --sudo inplace -w'inplace_password' -u 'Alex R' login -s 130.83.98.66
[*] then the actual import is started with:
/home/omerouser/OMERO.server/bin/omero import --bulk /OMERO/ManagedRepository/bulki.yml

This seams to work fine so far (with all your help), with the only limitation, that I need to produce the "user list" (for the dropdown) manually, since the script is in python 3 and I can not use the python 2 omero modules.

The problem I want to solve now is the following: Several users could issue the import command at the same time. So I added a random ID to the filelist/yml files as well as then matching to the CLI command for import. But then, when the import is finished I want the files deleted from the OMERO/ManagedRepositories to avoid chunk. Therefore I am looking for a trigger, when the import is finished and the yml/tsv files are no longer needed.
I hope this explains the bigger picture and I hope I am not doing something completely stupid.
Best wishes
Alex