We're Hiring!

Importing MATLAB/NumPy arrays into OMERO

General user discussion about using the OMERO platform to its fullest. Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

Importing MATLAB/NumPy arrays into OMERO

Postby clays » Mon Jul 24, 2017 7:40 am

Hello,

I would like to import raw sensor data and metadata acquired in MATLAB into OMERO. What is the best procedure for this? What I've gathered thus far:

Convert my raw data (2-d or 3-d vector) to a Java array and then use OMERO.tables for import. This is then associated with an image (how?). My metadata for the raw data would then just serve as the metadata for some dummy image. Am I on the right track?
Last edited by clays on Wed Jul 26, 2017 5:23 pm, edited 1 time in total.
clays
 
Posts: 8
Joined: Sun Jul 16, 2017 4:12 pm

Re: Importing MATLAB arrays and struct data types into OMERO

Postby manics » Mon Jul 24, 2017 9:04 am

Hi Clays

The OMERO MATLAB package has some support for OMERO.tables, though you'd have to reformat it from a matrix into a columnar table. There's an example at https://www-legacy.openmicroscopy.org/s ... ero-tables

For the metadata key-value pairs could be an alternative: https://www-legacy.openmicroscopy.org/s ... nnotations

How are you planning to use the sensor and metadata once it's in OMERO?

Simon
User avatar
manics
Team Member
 
Posts: 261
Joined: Mon Oct 08, 2012 11:01 am
Location: Dundee

Re: Importing MATLAB arrays and struct data types into OMERO

Postby clays » Tue Jul 25, 2017 9:29 pm

Thanks for the response Simon. Indeed, I've been looking at that example, and I've found myself a bit confused on how to use the approach there to be honest. I suppose it would be helpful if the example started out with a plain matlab matrix and then proceeded to go through the steps of how one would upload it into a table. For example, is using javaArray.m necessary? As for the metadata, that's what I meant I suppose. Does the table itself have the capacity to be associated with metadata? Does the table get associated with a "Dataset" (which, as far as i can tell, is strictly limited to images), and should my metadata then be the annotations for this dataset?

As for what I (we) would like to do with it, just use it for storage more or less. We are an imaging group, and so an image database is very useful of course. But our images are derived from sensor data that we acquire from an experiment. So ideally, it would be nice to upload the raw data after an experiment along with any metadata. Then we'd retrieve it later on, generate some images and more processed data, then send that back as a different dataset. And, while I'm aware that OMERO is primarily intended for microscopy images and not sensor data, if it smartly had the capacity for both, I think it would be even more powerful then it currently is. OCT, ultrasound, etc - all come into play.

So any tips would be much appreciated. We could of course store sensor data outside of OMERO, but having a unified platform is important I think to make things simpler.

Clay
Last edited by clays on Wed Jul 26, 2017 5:22 pm, edited 1 time in total.
clays
 
Posts: 8
Joined: Sun Jul 16, 2017 4:12 pm

Re: Importing MATLAB arrays and struct data types into OMERO

Postby clays » Wed Jul 26, 2017 12:25 pm

Also, it would be great if the same type of example could be done with NumPy arrays.
clays
 
Posts: 8
Joined: Sun Jul 16, 2017 4:12 pm

Re: Importing MATLAB arrays and struct data types into OMERO

Postby i.munro » Wed Jul 26, 2017 1:48 pm

Hi Clays

If it's just a matter of storage it is possible to store non-image(e'g. txt) files as attachments to images in OMERO.

Best Wishes

Ian
i.munro
 
Posts: 50
Joined: Thu Apr 25, 2013 1:01 pm

Re: Importing MATLAB arrays and struct data types into OMERO

Postby clays » Wed Jul 26, 2017 5:14 pm

It would be nice to be able to look at and process the data in the database later on. So using tables would be ideal.
clays
 
Posts: 8
Joined: Sun Jul 16, 2017 4:12 pm

Re: Importing MATLAB/NumPy arrays into OMERO

Postby wmoore » Mon Jul 31, 2017 1:12 pm

Hi,

OMERO.tables is really an API for creating and querying hdf files in OMERO. These tables are not part of the OMERO relational database, although that database is used to store a reference to the OMERO.tables files.

You can use annotations of various types to store any metadata that you have. These include Comment/text, File Annotation, Tag, Map Annotation (ordered list of Key/Value pairs), Long Annotation, etc. You can link annotations to Images or Datasets or even other Annotations.

So, you could create a Dataset and link all the raw table data to that Dataset.
Then subsequently retrieve this data and create images in this Dataset (or create a new one if desired).

Rows in an OMERO.table can be assigned to Images by using an "omero.grid.ImageColumn" instead of a LongColumn for Image IDs, and by convention we attach the OMERO.table as a File Annotation on the parent Dataset. However, there's no sql query that can tell you if there are any tables that have rows relating to a particular Image.

There's a python example at https://docs.openmicroscopy.org/omero/5 ... ero-tables which could be converted to using numpy arrays. Let us know if you need help with this.

Hope that helps,

Will.

PS: I'll refer your Matlab question to others' who have more experience to that...
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: Importing MATLAB/NumPy arrays into OMERO

Postby wmoore » Wed Aug 02, 2017 9:21 am

Another thought occurred to me:

If your raw sensor data is 2D or 3D arrays of data, this sounds like it should be stored in OMERO as an Image instead of in OMERO.tables?

E.g. an example of creating a new Image in OMERO from a series of numpy arrays can be found at
https://docs.openmicroscopy.org/omero/5 ... eate-image

You can read the Image pixel data back as a numpy array as shown https://docs.openmicroscopy.org/omero/5 ... ata-access

This could also be done in Java/Matlab.

Regards,

Will.
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: Importing MATLAB/NumPy arrays into OMERO

Postby clays » Fri Aug 04, 2017 11:35 am

Hey Will,

Thanks for your input. A couple things:

"Rows in an OMERO.table can be assigned to Images by using an 'omero.grid.ImageColumn' instead of a LongColumn for Image IDs, and by convention we attach the OMERO.table as a File Annotation on the parent Dataset. However, there's no sql query that can tell you if there are any tables that have rows relating to a particular Image."

I'm a bit confused by this. My notion was to store the raw sensor data for a frame in a table, where the row would correspond to one dimension and the column, the other. I'm guessing then, for a 3-D scan, I'd have to link multiple tables to a dataset with each one corresponding to a frame.

Regarding your second suggestion - just storing the raw data in image files - this is making more and more sense to me. Because the sensor data already has the dimensionality of an image, I suppose one could just consider it image data already. And although the images themselves may not be visually interesting on first glance, others have indicated that this approach can be useful in other ways. For example, I can see utility in merely hovering over a pixel with the mouse corresponding to some point in space and seeing the raw data value. Treating sensor data in the same way as image data also just seems like a simpler way to use the platform.
clays
 
Posts: 8
Joined: Sun Jul 16, 2017 4:12 pm

Re: Importing MATLAB/NumPy arrays into OMERO

Postby manics » Fri Aug 04, 2017 12:01 pm

Hi Clays

It sounds like storing your sensor data as a multi-dimensional image is the best option. OMERO.tables uses the underlying HDF5 table API (which is not the same as a HDF5 array). It's designed for "columnar" data along the lines of a typical CSV file, or R or Python Pandas dataframe.

Hope this helps, Simon
User avatar
manics
Team Member
 
Posts: 261
Joined: Mon Oct 08, 2012 11:01 am
Location: Dundee

Next

Return to User Discussion

Who is online

Users browsing this forum: Google [Bot] and 1 guest

cron