We're Hiring!

importing OME.tiffs in Omero 5

General and open developer discussion about using OMERO APIs from C++, Java, Python, Matlab and more! Please new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

If you are having trouble with custom code, please provide a link to a public repository, ideally GitHub.

importing OME.tiffs in Omero 5

Postby yuriy_alexandrov » Tue Mar 04, 2014 4:13 pm

Hi All,

This is in continuation to the OME.tiff managing activity - remind, I create them using OmeroMatlab bfsave-type methods, where the "Description" field is used to store additional metadata (encoded in UUIDFileNames) - and that was tested in Omero 4.4.10.

After updating to Omero 5 today, I found that when importing these files by 5-th version of Insight,
1) the description isn't set up and
2) for the larger file (360 planes, 1.44M), import took more than 40 minutes (that was found occasionally!).

There were no such problems in 4.4.10, - the description field was set up properly and importing the same large file took only 70 seconds.

I wonder if that is some sort of real Omero 5 glitch, or maybe I'm doing something wrong (maybe Bioformats version?).

Thanks,
Y.
Last edited by yuriy_alexandrov on Tue Mar 04, 2014 4:37 pm, edited 1 time in total.
yuriy_alexandrov
 
Posts: 126
Joined: Thu Oct 25, 2012 2:06 pm

Re: importing OME.tiffs in Omero 5

Postby wmoore » Tue Mar 04, 2014 4:22 pm

I don't have a solution, but this seems very similar to this bug: https://trac.openmicroscopy.org.uk/ome/ticket/11850
Although that was for OME-tiffs generated from OMERO export (and was fixed by OMERO 5.0.0), so I'm not sure if it's the same issue.

Could you send us a sample file - upload at
http://qa.openmicroscopy.org.uk/qa/upload/
and we can take a closer look.

Thanks,

Will.
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: importing OME.tiffs in Omero 5

Postby yuriy_alexandrov » Tue Mar 04, 2014 4:55 pm

Many thanks Will for fast reply,

I believe I sent the 1.44M file (at least got an automatic confirmation).

The proc that saves these OME.tiffs is here:
https://github.com/openmicroscopy/Imper ... _OMEtiff.m

Best,
Y.
yuriy_alexandrov
 
Posts: 126
Joined: Thu Oct 25, 2012 2:06 pm

Re: importing OME.tiffs in Omero 5

Postby wmoore » Fri Mar 07, 2014 11:00 am

Hi,

Thanks for sending the files and apologies for the delay.

I tried importing them and saw similar results as you. No description for either image and much slower import than expected for images of this size.

Command line import took about 8 minutes for the 1.4M OME.tiff:

Code: Select all
...
2014-03-07 10:12:27,883 32065      [l.Client-0] INFO      ome.formats.importer.cli.ErrorHandler - Number of errors: 0
2014-03-07 10:12:27,887 32069      [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2014-03-07 10:12:27,898 32080      [      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.1.0-demo-nottingham-DEV-ice35
2014-03-07 10:12:27,898 32080      [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 5.1.0-demo-nottingham-DEV-ice35 revision: faf44e1 date: 10 February 2014
2014-03-07 10:12:28,384 32566      [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2014-03-07 10:12:28,387 32569      [      main] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /Users/will/Documents/biology-data/from_QA_apache_repo/7942/fluor.OME.tiff...
2014-03-07 10:12:28,436 32618      [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /Users/will/Documents/biology-data/from_QA_apache_repo/7942/fluor.OME.tiff
2014-03-07 10:12:38,038 42220      [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /Users/will/Documents/biology-data/from_QA_apache_repo/7942/fluor.OME.tiff
2014-03-07 10:12:39,501 43683      [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2014-03-07 10:12:42,076 46258      [l.Client-2] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5  Logfile: 4207
2014-03-07 10:16:29,067 273249     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 4207
2014-03-07 10:20:13,191 497373     [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5  Logfile: 4207
2014-03-07 10:20:13,235 497417     [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 4207
2014-03-07 10:20:13,266 497448     [l.Client-2] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 4207
2014-03-07 10:20:13,401 497583     [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /Users/will/Documents/biology-data/from_QA_apache_repo/7942/fluor.OME.tiff



Inspecting the log via

Code: Select all
$ bin/omero download 4207 -


Showed a very large number of these lines:

Code: Select all
2014-03-07 10:13:00,424 DEBUG [    o.s.blitz.repo.ManagedImportRequestI] (3-thread-5) Plane:26 X:0 Y:37 TileWidth:430 TileHeight:1 arrayBuf.length:860


which I think is basically the same problem as the bug I mentioned above. The 'tiles' of pixels being read a quite small, so there's a very large number of tiles to read.
I'll re-open the ticket, but Melissa might re-assign if this is a different issue.


When I open the fluor.OME.tif in Fiji, the OMX-XML shows the Description as having an entire OME-XML block in it:

Code: Select all
<Description><?xml version="1.0" encoding="utf-8"?><OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2013-06" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2013-06 http://www.openmicroscopy.org/Schemas/OME/2013-06/ome.xsd"><Image ID="Image:0"><Pixels BigEndian="true" DimensionOrder="XYZCT" ..... </Pixels></Image></OME></Description>


Is this what you expect?

Cheers,

Will.
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: importing OME.tiffs in Omero 5

Postby yuriy_alexandrov » Mon Mar 10, 2014 12:33 pm

Hi Will,

Yes the Image Description contains whole XML block as I didn't find other way to retrieve metadata I need.
Please note that UUID FIlenames that are in that Description block (correct ones) are not reproduced by XML generated in the course of Insight import.

Best,
Y.
yuriy_alexandrov
 
Posts: 126
Joined: Thu Oct 25, 2012 2:06 pm

Re: importing OME.tiffs in Omero 5

Postby wmoore » Tue Mar 11, 2014 1:50 pm

I have created a ticket to address the loss of XML from OME.tiff description on import: https://trac.openmicroscopy.org.uk/ome/ticket/12056

If we can help with any work-around in the meantime, let us know.

Regards,

Will.
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: importing OME.tiffs in Omero 5

Postby wmoore » Mon Mar 31, 2014 3:15 pm

Hi Yuriy,

A fix for the slow import of your ome-tiffs has just been merged. They seemed to import within a few secs when I tested it - see https://github.com/openmicroscopy/bioformats/pull/1009.
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: importing OME.tiffs in Omero 5

Postby yuriy_alexandrov » Thu Apr 24, 2014 11:01 am

Hi again,

Question is (likely to Bioformats team) about possible changes.
I updated Bioformats to the new version, - that needed replacing "loci_tools.jar" by the "bioformats_package.jar" archive.

When tried to run the OME-tiff-saving code, I've got an error on the line

Code: Select all
img_description = char(loci.formats.MetadataTools.getOMEXML(metadata));                                                     

the error message suggests that the function getOMEXML is missing

log4j:WARN No appenders could be found for logger (loci.common.services.ServiceFactory).
log4j:WARN Please initialize the log4j system properly.
The class loci.formats.MetadataTools has no property or method named 'getOMEXML'.

Also, the function "bfsave_with_description.m" is missing from the package, but I applied it before and found very useful.

When I switch back to the old version of Bioformats - everything works.
Well, not counting problems described above, but at least OME.tiffs are created..

Best,
Y.
yuriy_alexandrov
 
Posts: 126
Joined: Thu Oct 25, 2012 2:06 pm

Re: importing OME.tiffs in Omero 5

Postby jmoore » Thu Apr 24, 2014 11:25 am

Hi Yuriy,

the major change between `loci_tools.jar` and `bioformats_package.jar` was the version of logging that the required (log4j --> logback). Some versions of MATLAB have a hard dependency on log4j. If that's the case, then you can stick with the newer version of the `loci_tools.jar` jar. See http://downloads.openmicroscopy.org/bio-formats/5.0.1/artifacts/loci_tools.jar which is under the Components section.

Cheers,
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: importing OME.tiffs in Omero 5

Postby yuriy_alexandrov » Thu Apr 24, 2014 1:19 pm

Thanks Will, Josh for addressing these issues.

With the "loci_tools.jar" file that Josh suggested, the same error occurs - it looks like the function

loci.formats.MetadataTools.getOMEXML

missed or renamed (or moved to another archive?).

Best,
Y.
yuriy_alexandrov
 
Posts: 126
Joined: Thu Oct 25, 2012 2:06 pm

Next

Return to Developer Discussion

Who is online

Users browsing this forum: No registered users and 1 guest