OMERO Bulk import

[alexr]

A small update:
I fetched the stdout from the OMERO server and used that to trigger the removal of the transferred files, so this seams to work.
I get a stdout like this:
---\n- Fileset: 485\n Image: [675]\n---\n- Fileset: 486\n Image: [676]\n---\n- Fileset: 487\n Image: [677]\n---\n- Fileset: 488\n Image: [678]\n---\n- Fileset: 489\n Image: [679]\n---\n- Fileset: 490\n Image: [680]\n'

So this I can use.
Alex

[jmoore]

Here again I describe the bigger picture:
I have written a small python GUI that allows the user (no CLI lovers)...

Interesting. Keep us posted how that goes. Is it on GitHub?

I hope this explains the bigger picture and I hope I am not doing something completely stupid.

I definitely understand better now, but there may yet still be things that could be simplified down the road with existing code. Additionally, a bit farther down the road, we'd like to add more infrastructure for exactly what you're trying to achieve. Happy to keep you in the loop for that.

A small update:
I fetched the stdout ...

So this I can use.

Great, glad to hear it.
~Josh

[alexr]

Dear Josh,
thanks for all your help. I am not on Git yet, but I am happy to share. I have two more questions:
The yaml bulk import can assign a "Dataset", that is given in the "filelist.tsv file" and is defiend by the yaml column structure:
columns:
- target
- path

and the tsv files has for the target:
Dataset:name:TestDataSet

Can I also add project names into the target column? Or can I add a third column that specifies the target? I have not found the documentation for that.
Thanks Alex

[jmoore]

Can I also add project names into the target column?

No, the CLI import in general doesn't support that yet. You will need to create any projects either before or after the import, putting the correct datasets into the correct project(s).

Or can I add a third column that specifies the target?

You mean the Project target? If so, no, the same as above.

I have not found the documentation for that.

Sorry about that. Documentation for the bulk import feature is certainly a work-in-progress.

~Josh

[alexr]

Ok thanks for the info.

I found one more issue, that is:
when I run:

Code: Select all

conn = BlitzGateway('user','passowrd',host='130.83.98.66', port=4064)
conn.connect()


I get :

Code: Select all

No handlers could be found for logger "omero.gateway"


But everything works fine. Can I just ignore the message?
Thanks
Alex

[jmoore]

Code: Select all

Can I just ignore the message?


In this particular case, yes, but it's probably better to go ahead and configure logging for your environment. If you don't have any preferences, you can add this to the top of your Python script(s):

Code: Select all

import logging
logging.basicConfig()


This will likely then show:

Code: Select all

ERROR:omero.gateway:No Pillow installed, line plots and split channel will fail!


in which case you can use:

Code: Select all

pip install Image


which will make the ERROR go away.

Cheers,
~Josh

[alexr]

Hi josh,
your fix solved the problem, but the pip needs to be on the client side, correct?

Now I am facing a new problem:
I try to get two drop down menues with the available groups and users. This works for all groups and the users of the user that opens the connection. But I can not get the users in differnt groups.
I tried:
Open a connection to OMERO as OMERO-admin

Code: Select all

conn = BlitzGateway('admin','adminpw',host='130.83.98.66', port=4064)
conn.connect()
for g in conn.listGroups():
   group_list.append(g.getName())


this works and I get a drop down menu.
--> select on group by drop down

change the connection environment:

Code: Select all

conn.SERVICE_OPTS.setOmeroGroup(groupID)  # Here is the problem, since I guess I need the ID of the group rather than the name.
group = conn.getGroupFromContext()
owners, members = group.getName()
for m in members:
   user_lsit.append(m.getOmeName())


--> make second drop dwon of users in selected group

Is there a way how I can get the IDs of a group with a given name or all IDs, or can I achieve my task completely differently?
Alex

[jmoore]

Hi josh,
your fix solved the problem, but the pip needs to be on the client side, correct?

Wherever you want to use OMERO.py, yes.

Now I am facing a new problem:

Ok. As you get a handle on all the moving parts, separate issues might be easier for others to digest, though.

Is there a way how I can get the IDs of a group with a given name or all IDs, or can I achieve my task completely differently?

Though Will or others may have a cleaner suggestion, one way would be:

Code: Select all

In [35]: for g in conn.listGroups():
    ...:     for l in g.getChildLinks():
    ...:         print g.name, "-->", l.child.omeName.val
    ...:
    ...:
system --> root
user --> root
user --> demo
user --> public
guest --> guest
Public --> demo
Public --> public


Cheers,
~Josh

[alexr]

Hi Josh,
your last suggestion for the users worked well. Thanks.

Now the script is working but I want to improve it. So far all connection details are hard coded and I wanted a more flexible solution. Therefore I added a settings page, where details are entered, such as:
omerouser (to run the commands, and ssh connect)
omerouserpw
omeroserverip
omeroserverport
inplace user
....

These entries are then stored into a dictionary and dumped using Pickle.

Then the main script imports these settings and I tried to replace the hard coded connection details. Here I faced a problem:
When I try to run the conn = BlitzGateway with variables I get an error and no connection.

So hard coded:

Code: Select all

conn = BlitzGateway('username','userpw',host='130.83.98.66', port=4064)
conn.connect()
ctx = conn.getEventContext()
print (ctx)
conn.close()


works well and I get:

Code: Select all

object #0 (::omero::sys::EventContext)
{
    shareId = -1
    sessionId = 3879
    sessionUuid = 68b3de8b-removed
    userId = 3
    userName = username
    sudoerId = <nil>
    sudoerName = <nil>
    groupId = 53
    groupName = user_group
    isAdmin = True
    adminPrivileges =
    {
        [0] = ReadSession
        [1] = DeleteScriptRepo
        [2] = ModifyUser
        [3] = WriteOwned
        [4] = WriteFile
        [5] = Sudo
        [6] = ModifyGroup
        [7] = DeleteManagedRepo
        [8] = ModifyGroupMembership
        [9] = WriteScriptRepo
        [10] = WriteManagedRepo
        [11] = DeleteFile
        [12] = DeleteOwned
        [13] = Chgrp
        [14] = Chown
    }
    eventId = -1
    eventType = Internal
    memberOfGroups =
    {
        [0] = 53
        [1] = 1
        [2] = 0
    }
    leaderOfGroups =
    {
        [0] = 53
    }
    groupPermissions = object #1 (::omero::model::Permissions)
    {
        _restrictions =
        {
        }
        _extendedRestrictions =
        {
        }
        _perm1 = -40
    }
}


But when I want to replace it with the variable retrieved from my stored dictionary:

Code: Select all

read_settings = pickle.load( open( "ipisettings.p", "rb" ) )


I can access all the values with e.g.:
read_settings["OMEROUSER"]
read_settings["OMEROIP"]
...

I checked the variable type and they are all strings.

So I tried to include them in the command like this:

Code: Select all

conn = BlitzGateway(read_settings["OMEROUSER"],read_settings["OMEROPW"],host = read_settings["OMEROIP"], port = read_settings["OMEROPORT"])
conn.connect()


But get the error:

Code: Select all

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/alexrapp/Documents/OMERO.server-5.4.5-ice36-b83/lib/python/omero/gateway/__init__.py", line 1560, in __init__
    self._resetOmeroClient()
  File "/Users/alexrapp/Documents/OMERO.server-5.4.5-ice36-b83/lib/python/omero/gateway/__init__.py", line 2085, in _resetOmeroClient
    args=['--Ice.Config='+','.join(self.ice_config)])
  File "/Users/alexrapp/Documents/OMERO.server-5.4.5-ice36-b83/lib/python/omero/__init__.py", line 67, in client
    return omero.clients.BaseClient(*args, **kwargs)
  File "/Users/alexrapp/Documents/OMERO.server-5.4.5-ice36-b83/lib/python/omero/clients.py", line 140, in __init__
    self._initData(id)
  File "/Users/alexrapp/Documents/OMERO.server-5.4.5-ice36-b83/lib/python/omero/clients.py", line 272, in _initData
    raise omero.ClientError(msg)
omero.ClientError: No host specified. Use omero.client(HOSTNAME), ICE_CONFIG, or similar.


I also tried to first pass the dictionary entries to variables and then pass them to the conn = BlitzGateway like this:
connUser = read_settings["OMERORUSER"]
connUser = read_settings["OMERORUSER"]
...
then :
conn = BlitzGateway(connUser, connPW, host = connIP, port = connPort)
conn.connect()
with no success.

Am I missing something here, or can I not pass variables to the BlitzConnect()?
Thanks
Alex

[jmoore]

I can't think of anything off-hand other than that your connIP field doesn't contain what you think.

Code: Select all

print connIP, type(connIP)


~Josh.