combining TIFF and CSV file with metadata

[jmoore]

Agreed! You have a tables log file now:

Code: Select all

Log files:  Tables-0.log                   3.0 KB 


Could you attach it to the thread?
~Josh

[Thorondor]

sure

Code: Select all


2012-05-07 13:18:59,646 INFO  [                       omero.util.Server] (MainThread) ********************************************************************************
2012-05-07 13:18:59,647 INFO  [                       omero.util.Server] (MainThread) Starting
2012-05-07 13:18:59,647 INFO  [                       omero.util.Server] (MainThread) Loaded dependency numpy (1.6.2rc1)
2012-05-07 13:18:59,647 INFO  [                       omero.util.Server] (MainThread) Loaded dependency tables (2.3.1, hdf=1.8.8)
2012-05-07 13:18:59,648 INFO  [                    omero.util.Resources] (Thread-2  ) Starting
2012-05-07 13:19:17,209 INFO  [                             omero.utils] (MainThread) Failed to get session on attempt 1
2012-05-07 13:19:27,428 INFO  [                       omero.util.Server] (MainThread) Entering main loop
2012-05-07 13:21:19,417 INFO  [                            omero.remote] (Dummy-3   )  Meth: TablesI.getTable
2012-05-07 13:21:19,417 INFO  [                    omero.tables.TablesI] (Dummy-3   ) getTable: 251 {}
2012-05-07 13:21:19,549 INFO  [                 omero.tables.HdfStorage] (Dummy-3   ) Size: 0 - Attaching Table-11D7C028-826A-495F-AF11-94ED0323B486 to /OMERO/Files/251
2012-05-07 13:21:19,550 INFO  [                            omero.remote] (Dummy-3   )  Rslt: 11D7C028-826A-495F-AF11-94ED0323B486 -t @ Tables-0.TablesAdapter
2012-05-07 13:21:19,579 INFO  [                            omero.remote] (Dummy-4   )  Meth: TableI.initialize
2012-05-07 13:21:19,595 INFO  [                     omero.tables.TableI] (Dummy-4   ) Initialized Table-11D7C028-826A-495F-AF11-94ED0323B486 with 2 col(s)
2012-05-07 13:21:19,595 INFO  [                            omero.remote] (Dummy-4   )  Rslt: None
2012-05-07 13:21:19,596 INFO  [                            omero.remote] (Dummy-5   )  Meth: TableI.addData
2012-05-07 13:21:19,597 INFO  [                     omero.tables.TableI] (Dummy-5   ) Added 10 row(s) of data to Table-11D7C028-826A-495F-AF11-94ED0323B486
2012-05-07 13:21:19,598 INFO  [                            omero.remote] (Dummy-5   )  Rslt: None
2012-05-07 13:21:19,598 INFO  [                            omero.remote] (Dummy-6   )  Meth: TableI.close
2012-05-07 13:21:19,599 INFO  [                 omero.tables.HdfStorage] (Dummy-6   ) Size: 1 - Detaching Table-11D7C028-826A-495F-AF11-94ED0323B486 from /OMERO/Files/251
2012-05-07 13:21:19,599 INFO  [                 omero.tables.HdfStorage] (Dummy-6   ) Cleaning storage: /OMERO/Files/251
2012-05-07 13:21:19,600 INFO  [                     omero.tables.TableI] (Dummy-6   ) Closed Table-11D7C028-826A-495F-AF11-94ED0323B486
2012-05-07 13:21:19,772 INFO  [                     omero.tables.TableI] (Dummy-6   ) Updated file object 251 to sha1=f48b6e69ebc9a55f01c9fcefc60f6eb4802e86ad (73792 bytes)
2012-05-07 13:21:19,772 INFO  [                            omero.remote] (Dummy-6   )  Rslt: None
2012-05-07 13:21:19,773 INFO  [                            omero.remote] (Dummy-7   )  Meth: TableI.getOriginalFile
2012-05-07 13:21:19,774 INFO  [                     omero.tables.TableI] (Dummy-7   ) Table-11D7C028-826A-495F-AF11-94ED0323B486.getOriginalFile() => id=251
2012-05-07 13:21:19,774 INFO  [                            omero.remote] (Dummy-7   )  Rslt: object #0 (::omero::model::OriginalFile)
{
    _id = object #1 (::omero::RLong)
    {
        _val = 251
    }
    _deta


[jmoore]

That certainly looks fine. Could you show us your whole tables.py script to see where it's going wrong?
~J

[Thorondor]

it is still the example you posted some pages ago:

Code: Select all


#!/usr/bin/env python
# -*- coding: utf-8 -*-

#
# Copyright (C) 2011 University of Dundee & Open Microscopy Environment.
#                    All Rights Reserved.
# Use is subject to license terms supplied in LICENSE.txt
#

"""
FOR TRAINING PURPOSES ONLY!
"""

import omero
import omero.grid
from omero.gateway import BlitzGateway
#from Connect_To_OMERO import USERNAME, PASSWORD, HOST, PORT

USERNAME = "*******"
PASSWORD = "********"
HOST = "localhost"
PORT = "4064"



# Create a connection
# =================================================================
conn = BlitzGateway(USERNAME, PASSWORD, host=HOST, port=PORT)
conn.connect()

print conn.connect()



# Configuration
# =================================================================
datasetId = 33


# Create a name for the Original File (should be unique)
# =================================================================
from random import random
tablename = "TablesDemo:%s" % str(random())

col1 = omero.grid.LongColumn('Uid', 'testLong', [])
col2 = omero.grid.StringColumn('MyStringColumnInit', '', 64, [])

columns = [col1, col2]


# Create and initialize a new table.
# =================================================================
repositoryId = 1
table = conn.c.sf.sharedResources().newTable(repositoryId, tablename)
table.initialize(columns)
#table.initialzis(WellColumn)


# Add data to the table.
# =================================================================
ids = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
strings = ["one", "two", "three", "four", "five",\
           "six", "seven", "eight", "nine", "ten"]
data1 = omero.grid.LongColumn('Uid', 'test Long', ids)
data2 = omero.grid.StringColumn('MyStringColumn', '', 64, strings)
data = [data1, data2]
table.addData(data)
table.close()           # when we're done, close.


# Get the table as an original file...
# =================================================================
orig_file = table.getOriginalFile()
orig_file_id = orig_file.id.val
# ...so you can attach this data to an object. E.g. Dataset
fileAnn = omero.model.FileAnnotationI()
fileAnn.setFile(omero.model.OriginalFileI(orig_file_id, False))     # use unloaded OriginalFileI
fileAnn = conn.getUpdateService().saveAndReturnObject(fileAnn)
link = omero.model.DatasetAnnotationLinkI()
link.setParent(omero.model.DatasetI(datasetId, False))
link.setChild(omero.model.FileAnnotationI(fileAnn.id.val, False))
conn.getUpdateService().saveAndReturnObject(link)


# Table API
# =================================================================
# See: http://hudson.openmicroscopy.org.uk/job/OMERO/javadoc/slice2html/omero/grid/Table.html
openTable = conn.c.sf.sharedResources().openTable(orig_file)

print "Table Columns:"
for col in openTable.getHeaders():
    print "   ", col.name

rowCount = openTable.getNumberOfRows()
print "Row count:", rowCount


# Get data from every column of the specified rows
# =================================================================
rowNumbers = [3, 5, 7]
print "\nGet All Data for rows: ", rowNumbers
data = openTable.readCoordinates(range(rowCount))
for col in data.columns:
    print "Data for Column: ", col.name
    for v in col.values:
        print "   ", v


# Get data from specified columns of specified rows
# =================================================================
colNumbers = [1]
start = 3
stop = 7
print "\nGet Data for cols: ", colNumbers,\
        " and between rows: ", start, "-", stop

data = openTable.read(colNumbers, start, stop)
for col in data.columns:
    print "Data for Column: ", col.name
    for v in col.values:
        print "   ", v


# Query the table for rows where the 'Uid' is in a particular range
# =================================================================
queryRows = openTable.getWhereList("(Uid > 2) & (Uid <= 8)",\
        variables={}, start=0, stop=rowCount, step=0)
data = openTable.readCoordinates(queryRows)
for col in data.columns:
    print "Query Results for Column: ", col.name
    for v in col.values:
        print "   ", v
openTable.close()           # we're done


# In future, to get the table back from Original File
# =================================================================
orig_table_file = conn.getObject("OriginalFile", attributes={'name': tablename})    # if name is unique
savedTable = conn.c.sf.sharedResources().openTable(orig_table_file._obj)
print "Opened table with row-count:", savedTable.getNumberOfRows()


# Close connection:
# =================================================================
# When you're done, close the session to free up server resources.
conn._closeSession()


[jmoore]

Did you configure the dataset id for your install and if so, what group is it in?

Code: Select all

    # Configuration
    # =================================================================
    datasetId = 33


~J.

[Thorondor]

ah, I thought it creates a new dataset. After creating one and setting the correct number, it looks much better:

Code: Select all


python tables.py
No handlers could be found for logger "blitz_gateway"                                                                                         
True                                                                                                                                           
Table Columns:
    Uid                                                                                                                                                     
    MyStringColumnInit                                                                                                                                     
Row count: 10                                                                                                                                               

Get All Data for rows:  [3, 5, 7]
Data for Column:  Uid                                                                                                                                                     
    1                                                                                                                                                                     
    2                                                                                                                                                                     
    3
    4                                                                                                                                                                             
    5                                                                                                                                                                             
    6
    7                                                                                                                                                                             
    8                                                                                                                                                                             
    9                                                                                                                                                                             
    10                                                                                                                                                                           
Data for Column:  MyStringColumnInit
    one
    two                                                                                                                                                                           
    three
    four                                                                                                                                                                         
    five                                                                                                                                                                         
    six
    seven
    eight
    nine
    ten

Get Data for cols:  [1]  and between rows:  3 - 7
Data for Column:  MyStringColumnInit
    four
    five
    six
    seven
Traceback (most recent call last):
  File "tables.py", line 127, in ?
    variables={}, start=0, stop=rowCount, step=0)
  File "/data/applications/OMERO.server-Beta-4.3.4/lib/python/omero_Tables_ice.py", line 687, in getWhereList
    return _M_omero.grid.Table._op_getWhereList.invoke(self, ((condition, variables, start, stop, step), _ctx))
omero.InternalException: exception ::omero::InternalException
{
    serverStackTrace = Traceback (most recent call last):
  File "/omero/OMERO.server-Beta-4.3.4/lib/python/omero/util/decorators.py", line 61, in exc_handler
  File "/omero/OMERO.server-Beta-4.3.4/lib/python/omero/util/decorators.py", line 28, in handler
  File "/data/applications/OMERO.server-Beta-4.3.4/lib/python/omero/tables.py", line 601, in getWhereList
    rv = self.storage.getWhereList(self.stamp, condition, variables, None, start, stop, step)
  File "/omero/OMERO.server-Beta-4.3.4/lib/python/omero/util/decorators.py", line 81, in with_lock
  File "/data/applications/OMERO.server-Beta-4.3.4/lib/python/omero/tables.py", line 65, in check_and_update_stamp
    return func(*args, **kwargs)
  File "/data/applications/OMERO.server-Beta-4.3.4/lib/python/omero/tables.py", line 394, in getWhereList
    return self.__mea.getWhereList(condition, variables, None, start, stop, step).tolist()
  File "/usr/lib64/python2.4/site-packages/tables/table.py", line 1541, in getWhereList
    coords = [ p.nrow for p in
  File "/usr/lib64/python2.4/site-packages/tables/table.py", line 1437, in _where
    compiled = self._compileCondition(condition, condvars)
  File "/usr/lib64/python2.4/site-packages/tables/table.py", line 1316, in _compileCondition
    compiled = compile_condition(condition, typemap, indexedcols, copycols)
  File "/usr/lib64/python2.4/site-packages/tables/conditions.py", line 391, in compile_condition
    func = NumExpr(expr, signature, copy_args=copycols)
  File "/usr/lib64/python2.4/site-packages/numexpr-1.4.2-py2.4-linux-x86_64.egg/numexpr/necompiler.py", line 518, in NumExpr
    input_names)
RuntimeError: invalid program: reduction operations must occur last

    serverExceptionClass =
    message = Internal exception
}


[jmoore]

Unfortunately, we've once again run into a weird situation. Your numpy is much newer than mine:

Code: Select all

2012-05-07 15:25:25,126 INFO  [                       omero.util.Server] (MainThread) Loaded dependency numpy (1.3.0)
2012-05-07 15:25:25,126 INFO  [                       omero.util.Server] (MainThread) Loaded dependency tables (2.1.2, hdf=1.8.4)


Any chance of your rolling back somewhat?
~Josh

P.S. I filed a pytables ticket.

[jmoore]

Can you tell me what version of numexpr you have installed?

Code: Select all

python -c "import numexpr; print numexpr.__file__"
python -c "import numexpr.version; print numexpr.version.version"


Thanks,
~Josh

[Thorondor]

these python issues makes me go like:

now I tired to install a older numpy version and I get:

Code: Select all

import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib64/python2.4/site-packages/numpy/__init__.py", line 128, in ?
    from version import git_revision as __git_revision__


trying to fix that, what numexp version would you recommend?
the newest one?
http://code.google.com/p/numexpr/downloads/detail?name=numexpr-2.0.1.tar.gz&can=2&q=

[jmoore]

now I tired to install a older numpy version and I get:

How did you try to install?

trying to fix that, what numexp version would you recommend?
the newest one?

I have 1.4.2, but that's not to say that that will work with your other version issues.

Cheers,
~Josh