We're Hiring!

Populate_Metadata script in 5.3.3

General user discussion about using the OMERO platform to its fullest.
Please note:
There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

Populate_Metadata script in 5.3.3

Postby dsudar » Thu Sep 14, 2017 12:29 am

Hi team,
I was trying the Populate_Metadata.py script on a 384 well plate with a csv file that starts:
Code: Select all
Well,Compound,Concentration
A01,Media,0
A02,Media,0
A03,Media,0
A04,Media,0
A05,Media,0
A06,Media,0
A07,Media,0
A08,Media,0
A09,Media,0
A10,Media,0
A11,Media,0
A12,Media,0
A13,Media,0
A14,Media,0
A15,Media,0
A16,Media,0
A17,Media,0
A18,Media,0
A19,Media,0
A20,Media,0
A21,Media,0
A22,Media,0
A23,Media,0
A24,Media,0
B01,Media,0
B02,leucovorin,20000
B03,Panotinib,137.17
B04,Panotinib,45.72
B05,Pertuzumab+Herceptin,1.23
B06,Oxalipatin,740.74
B07,Lenvatinib,1234.57
etc....


The errors I get say:
Code: Select all
WARNING:omero.util.populate_metadata:Skipping table row 0! Missing well row or column for well name population!
Traceback (most recent call last):
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/populate_metadata.py", line 912, in post_process
    v = self.value_resolver.get_well_name(well_id, plate)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/populate_metadata.py", line 312, in get_well_name
    well = self.wrapper.get_well_by_id(well_id, plate)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/populate_metadata.py", line 578, in get_well_by_id
    return wells[well_id]
KeyError: -1L
WARNING:omero.util.populate_metadata:Skipping table row 1! Missing well row or column for well name population!
Traceback (most recent call last):
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/populate_metadata.py", line 912, in post_process
    v = self.value_resolver.get_well_name(well_id, plate)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/populate_metadata.py", line 312, in get_well_name
    well = self.wrapper.get_well_by_id(well_id, plate)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/populate_metadata.py", line 578, in get_well_by_id
    return wells[well_id]
KeyError: -1L
WARNING:omero.util.populate_metadata:Skipping table row 2! Missing well row or column for well name population!
Traceback (most recent call last):
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/populate_metadata.py", line 912, in post_process
    v = self.value_resolver.get_well_name(well_id, plate)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/populate_metadata.py", line 312, in get_well_name
    well = self.wrapper.get_well_by_id(well_id, plate)
  File "/usr/local/OMERO/OMERO.server/lib/python/omero/util/populate_metadata.py", line 578, in get_well_by_id
    return wells[well_id]
KeyError: -1L

etc....


On another plate using the exact same csv annotation file, it doesn't give errors but nothing appears in the Tables tab for the wells or images in each well. For both plates, a bulk_annotations attachment does get created.

I believe this all worked fine under 5.2.8 and I recently upgraded both my production servers to 5.3.3 (and will upgrade to 5.3.4 shortly). Am I doing something wrong or is something different between 5.2.x and the 5.3.3?

Thanks,
- Damir
dsudar
 
Posts: 169
Joined: Mon May 14, 2012 8:43 pm
Location: Berkeley, CA, USA

Re: Populate_Metadata script in 5.3.3

Postby dsudar » Thu Sep 14, 2017 4:47 am

Hi team,

Quick update on my query: turns out that I apparently had an unfortunate and incompatible combination of versions of PyTables, HDF5, numexpr, and/or numpy installed. Especially when installing and upgrading with pip, it is pretty easy to get into such a situation and there's not much that lets you know. Any thoughts how to check you have something compatible?

Anyway, by downgrading some of those pip packages, I was able to get the Populate_Metadata.py script to work correctly again.

Thanks,
- Damir
dsudar
 
Posts: 169
Joined: Mon May 14, 2012 8:43 pm
Location: Berkeley, CA, USA

Re: Populate_Metadata script in 5.3.3

Postby manics » Thu Sep 14, 2017 8:29 am

Hi Damir

What operating system are you using? We've had problems with pytables/numpy/numexpr etc in the past. With recent distributions (e.g. CentOS 7) we recommend using the distribution packages where possible, and only using pip to install packages which are too old or non-existent.

Simon
User avatar
manics
Team Member
 
Posts: 210
Joined: Mon Oct 08, 2012 11:01 am
Location: Dundee

Re: Populate_Metadata script in 5.3.3

Postby manics » Thu Sep 14, 2017 8:40 am

I've just had another though. pytables should have a built in test method:
Code: Select all
import tables
tables.test()

If most of the tests pass it's probably working.
User avatar
manics
Team Member
 
Posts: 210
Joined: Mon Oct 08, 2012 11:01 am
Location: Dundee

Re: Populate_Metadata script in 5.3.3

Postby dsudar » Thu Sep 14, 2017 6:12 pm

Hi Simon,

This is on Ubuntu 14.04 LTS. This server has been through many upgrades starting with being Ubuntu 12.04 LTS and running OMERO 4.4.5. So it can probably do with some cleaning up and indeed there has been some mix of apt-get installed packages and pip-installed ones. On my list of to-dos.

I actually did run the tables.test() and that passed even while the Populate_Metadata.py script was not doing what it should do. I'll have a close look at the Tables.log and Blitz.log to see if I find any clues.

Cheers,
- Damir
dsudar
 
Posts: 169
Joined: Mon May 14, 2012 8:43 pm
Location: Berkeley, CA, USA


Return to User Discussion

Who is online

Users browsing this forum: No registered users and 1 guest