importing OME.tiffs in Omero 5

[wmoore]

Hi Yuriy,

If you want to get back the image that you originally imported, you should always be able to Download it from Insight. Look at the download options at the top of the right panel.

However, we'd also like to fix the OME-TIFF export bug, so if you could upload an example of the file you imported to http://qa.openmicroscopy.org.uk/qa/upload/ that would be great, thanks.

Will.

[yuriy_alexandrov]

Clarification to the previous - it is actually not that bad.
The OME-tiff Export option works OK for usual images and for the data upoladed from our created OME-tiffs (OPT data).

It seems like, the problem is when Modulo FLIM annotation is added to the metadata.
If the same OME-tiff files are created without Modulo annotation - then after uploading to Omero, the Insight's "Save as OME-tiff" option works normally on uploaded data (but they become unusable for FLIMfit certainly).

Best,
Y.

[yuriy_alexandrov]

Many thanks Will,

I uploaded couple of these OME-tiffs that have no problems in both Omero and FLIM-fit, but can't be "Exported".
I tried to use Omero metadata viewer on them and it also fails.

Best regards,
Y.

[wmoore]

I have created a ticket for OME-TIFF generation at https://trac.openmicroscopy.org.uk/ome/ticket/12313

But please remember that if you simply want to get back the OME-TIFF that you imported, you can use the Download option from Insight, which works fine.

I also see that there is no Original Metadata displayed. Can you tell us what you'd expect to see in the Original Metadata list?

[yuriy_alexandrov]

Great - the "Download" button solves the problem!
I just missed this option, was so focused on "Export".


Many thanks for help everybody.

Best,
Y.

[yuriy_alexandrov]

By the results of discussion in Paris, the proper place for custom XML metadata and for saving XY-plane filenames (as they go from camera) were proposed (thanks Andy, Melissa).

Custom XML metadata are read from the disk file (it is important to skip the first line - thanks Sebastien)

Code: Select all

    fid = fopen([folder filesep xmlfilename],'r');
    fgetl(fid);
    description = fscanf(fid,'%c');           
    fclose(fid);

and added to metadata as XMLAnnotation

Code: Select all

    metadata.setXMLAnnotationID('Annotation:0',0);
    metadata.setXMLAnnotationValue(description,0);   


this annotation might be then retrieved by the command (r is the reader)

Code: Select all

OMEXMLdescription = r.getMetadataStore().getXMLAnnotationValue(0);

By that, the problem with custom XML looks like, resolved, - it is saved and retrieved without distortions.

To store camera output XY-plane filenames, the following block of code has been proposed:

Code: Select all

for i = 1:num_files
...
metadata.setCommentAnnotationID('Annotation:0',i-1);
metadata.setCommentAnnotationValue(char(file_names{i}),i-1);
metadata.setPlaneTheZ(toNNI(z-1),0,i-1);
metadata.setPlaneTheC(toNNI(c-1),0,i-1);
metadata.setPlaneTheT(toNNI(t-1),0,i-1);
metadata.setPlaneAnnotationRef('Annotation:0',0,i-1,0); 
...
end

these annotations are then retrieved by the command (r is the reader)

Code: Select all

..r.getMetadataStore().getCommentAnnotationValue(i)..

This block however, introduces problems to OME-tiff.
First, the attempt of importing OME-tiff created with this option to Omero - fails.
Second, the lifetime OME-tiff (with ModuloAlong according convention) created with this option, becomes unreadable by Ian's new Bioformats FLIMfit. Likely by the same reason.
So for now it isn't clear if it is workable.

Best,
Y.

[sbesson]

HI Yuriy,

I had a look at your OME-TIFFs and they do fail the XML validation check via Bio-Formats xmlvalid tool. If you want to stay in MATLAB, the equivalent of xmlvalid should be the following:

Code: Select all

xmlValidate = loci.formats.tools.XMLValidate();
filename = '/Users/sebastien/Desktop/9280/A-1-XY_00000.OME.tiff';
comment = loci.formats.tiff.TiffParser(filename).getComment()
xmlValidate.process(filename, java.io.BufferedReader(java.io.StringReader(comment)));


Before performing any import,I would make sure the basic XML validation passes.

Additionally, looking at your code, I am not sure it will produce what you expect. For instance:

Code: Select all

metadata.setCommentAnnotationID('Annotation:0',i-1);


creates a series of annotations having exactly the same IDs `Annotation:0`.
Also, I am not sure about the plane annotation reference since

Code: Select all

metadata.setPlaneAnnotationRef('Annotation:0',0,i-1,0); 


is not iterating over the annotationRefIndex but rather always use 0.

Cheers,
Sebastien

[yuriy_alexandrov]

Hi Sebastien,

I checked what is stored in CommentAnnotation (this is for our OPT file but the same code is used), and it is as expected -

>> r = bfGetReader('c:\users\yalexand\omero_ya\fluor.ome.tiff');
>> r.getMetadataStore().getCommentAnnotationValue(0)

ans =

fr000Rot000_0000.tif

>> r.getMetadataStore().getCommentAnnotationValue(1)

ans =

fr000Rot001_0000.tif

>> r.getMetadataStore().getCommentAnnotationValue(359)

ans =

fr000Rot359_0000.tif

These are the values that were written.

On "xmlvalid" - I would like not to misunderstand.
I first thought that the comments contained some inappropriate syntax.
So I tried to replace the annotations for an arbitrary strings like 'simple annotation'. That didn't fix problem.
Maybe one can switch from CommentAnnotation to another type?
It is really desirable if we save these filenames, as they might include some residual metadata.

So I'm still not sure if
1) If one uses CommentAnnotation, then "xmlvalid" will fail (not good)
or
2) after calling "xmlvalid" the problem will be resolved (slim chance)

Thanks,
Y.

[sbesson]

Hi Yuriy,

There are obviously other solutions for storing extra metadata but using CommentAnnotation is a valid one in the sense that it should work with the OME model.

With regard to your import failure, the error does not come from the value that is stored in the CommentAnnotation. As said in my previous reply, all the comment annotations in the file share exactly the same ID. This is an issue and is certainly the cause of the import failure:

Code: Select all

sebastien@jrs-macbookpro-25399:dist $ bin/omero import ~/Desktop/9280/B-1-XY_00001.OME.tiff
Previous session expired for root on localhost:4064
Server: [localhost]octopus.openmicroscopy.org
Username: [root]user-1
Password:
Created session 6315b156-0f52-419b-9520-98dd65dd3910 (user-1@octopus.openmicroscopy.org:4064). Idle timeout: 10.0 min. Current group: private-1
2014-06-12 14:36:40,175 5540       [      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.0.3-ice35-SNAPSHOT
2014-06-12 14:36:40,513 5878       [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 5.0.3-ice35-SNAPSHOT revision: 0c4215a date: 11 June 2014
...
Caused by: java.lang.RuntimeException: Unable to synchronize reference Annotation:0 --> omero.model.PlaneInfo:0:0
   at ome.formats.model.ReferenceProcessor.process(ReferenceProcessor.java:130)
   ... 34 more
}

   at ome.formats.importer.ImportLibrary$ImportCallback.onFinished(ImportLibrary.java:627) ~[blitz.jar:na]
   at omero.cmd.CmdCallbackI.finished(CmdCallbackI.java:301) [blitz.jar:na]
   at omero.cmd._CmdCallbackDisp.___finished(_CmdCallbackDisp.java:118) [blitz.jar:na]
   at omero.cmd._CmdCallbackDisp.__dispatch(_CmdCallbackDisp.java:145) [blitz.jar:na]
   at IceInternal.Incoming.invoke(Incoming.java:222) [ice.jar:na]
   at Ice.ConnectionI.invokeAll(ConnectionI.java:2482) [ice.jar:na]
   at Ice.ConnectionI.dispatch(ConnectionI.java:1258) [ice.jar:na]
   at Ice.ConnectionI.message(ConnectionI.java:1213) [ice.jar:na]
   at IceInternal.ThreadPool.run(ThreadPool.java:321) [ice.jar:na]
   at IceInternal.ThreadPool.access$300(ThreadPool.java:12) [ice.jar:na]
   at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:693) [ice.jar:na]
   at java.lang.Thread.run(Thread.java:695) [na:1.6.0_65]
2014-06-12 14:37:17,153 42518      [      main] ERROR        ome.formats.importer.ImportLibrary - Error on import
java.lang.Exception: Import failure
   at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:526) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary.importCandidates(ImportLibrary.java:271) ~[blitz.jar:na]
   at ome.formats.importer.cli.CommandLineImporter.start(CommandLineImporter.java:216) [blitz.jar:na]
   at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:721) [blitz.jar:na]
2014-06-12 14:37:17,153 42518      [      main] INFO         ome.formats.importer.ImportLibrary - Exiting on error



Additionally, xmlvalid warns you that your annotations are missing a namespace prefix. We will investigate why Bio-Formats does not add this prefix to the OME-XML. But this is not-related to the ID duplication error.

More generally, validating your OME-TIFF with xmlvalid is crucial for your application. An OME-TIFF passing this sanity check should import into OMERO. An OME-TIFF failing XML validation will likely cause import errors and possibly complete failures.

Best,
Sebastien & Andrew

[yuriy_alexandrov]

Thanks - I will certainly add "xmlvalid" check.

With this block of code - as I understand I should replace 'Annotation:0' for the real filename in both places and replace one of zeros at setPlaneAnnotationRef parameters for (i-1), but not sure which one.

Will this go?

Code: Select all

for i = 1:num_files
...
metadata.setCommentAnnotationID(char(file_names{i}),i-1);
metadata.setCommentAnnotationValue(char(file_names{i}),i-1);
metadata.setPlaneTheZ(toNNI(z-1),0,i-1);
metadata.setPlaneTheC(toNNI(c-1),0,i-1);
metadata.setPlaneTheT(toNNI(t-1),0,i-1);
metadata.setPlaneAnnotationRef(char(file_names{i}),0,i-1,i-1); 
...
end

Best,
Y.