We're Hiring!

new Olympus Scan^R filesets error on import

Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats
Please note:
Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats

If you are having trouble with image files, there is information about reporting bugs in the Bio-Formats documentation. Please send us the data and let us know what version of Bio-Formats you are using. For issues with your code, please provide a link to a public repository, ideally GitHub.

Re: new Olympus Scan^R filesets error on import

Postby dgault » Fri Aug 12, 2016 4:02 pm

Hi Damir,

What I know so far regarding this error is that the following is occurring:
- The file reader uses the experiment_descriptor to parse the experiment settings
- A list of the available tiff files is found from the data folder or immediate directory
- Any tiff files containing valid position, well, z, t and channel name are added to a list of associated tiffs
- The first of these tiffs is read
- At this point we are seeing an exception as it is unable to locate the first of these tiff files

As the file uploads appear to have been successful i am not sure as of yet why this particular file is causing problems. The fact that the file imports correctly through Fiji would lead me to believe the issue is not due to any issues with the parsing or the tiff files themselves but somehow related to accessing the tif file location.

I will continue to try and do some more in depth testing on my side to get to the bottom of this issue.
User avatar
dgault
Team Member
 
Posts: 208
Joined: Fri Aug 14, 2015 2:56 pm

Re: new Olympus Scan^R filesets error on import

Postby dsudar » Tue Aug 30, 2016 1:22 am

Hi David,

I just found a little time to look a bit more carefully at this issue where the older ScanR files (version 2.5.1) are imported into OMERO 5.2.5 without trouble and the newer ScanR files created with version 2.6.1 result in the exception we both see.

I compared line by line the experiment_descriptor.xml files and could not identify anything that could explain the issue. Sure, there are some differences but nothing that explains the issue. So indeed, I started looking at the tiff files themselves. In the 2.6.1-generated tiff files, there is a lot more metadata and specifically a bunch of supposed OME metadata. I enclose a screenshot of the metadata that Fiji's BioFormats importer extracted. It looks a bit like a jumble of stuff that may not even be appropriate OME but is there any chance that the OMERO reader gets very upset by that stuff? Might that explain the issue?

Thanks,
- Damir

ScanR_TIFF_Metadata.PNG
ScanR_TIFF_Metadata.PNG (26 KiB) Viewed 4324 times
dsudar
 
Posts: 235
Joined: Mon May 14, 2012 8:43 pm
Location: Berkeley, CA, USA

Re: new Olympus Scan^R filesets error on import

Postby dgault » Tue Aug 30, 2016 4:17 pm

Hi Damir,

I think I have finally managed to isolate the problem with this fileset. It appears that the tiff files are being uploaded as a separate fileset which is the cause of the exception we had been seeing.

The reason as to why they are being uploaded separately and not alongside the metadata appears to be because the ScanrReader first identifies a list of all the tiff files in the data directory, which it is doing so correctly, it then checks each one to see if it belongs to the fileset. It does this by checking a number of things, in this case the problem occurs when it reads the software tag from the IFD (tag 305). It is expecting the software tag to read 'National Instruments IMAQ'. Any tiff files with this tag are considered to be part of the fileset while those without the tag set are not.

So in short it is due to this software tag not being set of the tiff files that it is not correctly associating them with the xml and dat files.

David
User avatar
dgault
Team Member
 
Posts: 208
Joined: Fri Aug 14, 2015 2:56 pm

Re: new Olympus Scan^R filesets error on import

Postby dsudar » Tue Aug 30, 2016 6:09 pm

Hi David,

Great detective work. Yes, I guess that would make sense: the ScanR software is based on National Instruments's LabView and so having that IFD in there makes sense. However, it appears that with the new version of the software (we currently use 2.6.2 and are looking at upgrading to 2.7.1 when that comes out) the Olympus folks changed that. I would imagine that all future versions will have the new IFD (whatever it now is).

So would it be a fairly straightforward mod to the ScanR reader code to accept the new IFD as an alternative acceptable identifier for TIFFs that are part of the set?

Thanks,
- Damir
dsudar
 
Posts: 235
Joined: Mon May 14, 2012 8:43 pm
Location: Berkeley, CA, USA

Re: new Olympus Scan^R filesets error on import

Postby dgault » Wed Aug 31, 2016 9:04 am

It is good to find out that this has changed, we will certainly have to modify the reader to find a new way of associating these tiff files.

I have created a bug tracking ticket for this and will hopefully be able to work on putting a fix in place over the next days - https://trac.openmicroscopy.org/ome/ticket/13286#ticket
User avatar
dgault
Team Member
 
Posts: 208
Joined: Fri Aug 14, 2015 2:56 pm

Re: new Olympus Scan^R filesets error on import

Postby dsudar » Wed Sep 14, 2016 9:08 pm

Hi David,

We upgraded our ScanR acquisition system to the most current version of the Olympus ScanR software, version 2.7.0 and started acquiring some data with that. To check whether there were any differences between the previous version (2.6.1) and this one, I also tried to import a plate scanned with 2.7.0 and while the behavior was very similar, there are some differences that may need to be taken into account. Thus I uploaded to the QA system a very small truncated fileset from scanning a plate. The main difference in behavior I saw is that the import starts with transferring the tiff image files first (which then end up either in "Orphaned Images" or they disappear into thin air) and then when it tries to interpret the experiment_descriptor.xml file, it excepts with the same error that happened with the version 2.6.1 ScanR filesets. An excerpt from the import messages is:
2016-09-14 13:53:39,596 791 [ main] INFO ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2016-09-14 13:54:01,265 22460 [ main] INFO ome.formats.importer.ImportCandidates - 7 file(s) parsed into 2 group(s) with 2 call(s) to setId in 21654ms. (21670ms total) [0 unknowns]
2016-09-14 13:54:02,238 23433 [ main] INFO ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2016-09-14 13:54:02,835 24030 [ main] INFO ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2016-09-14 13:54:03,180 24375 [ main] INFO ome.formats.OMEROMetadataStoreClient - Server: 5.2.5
2016-09-14 13:54:03,180 24375 [ main] INFO ome.formats.OMEROMetadataStoreClient - Client: 5.2.5-ice35-b28
2016-09-14 13:54:03,180 24375 [ main] INFO ome.formats.OMEROMetadataStoreClient - Java Version: 1.7.0_101
2016-09-14 13:54:03,180 24375 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2016-09-14 13:54:03,180 24375 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2016-09-14 13:54:03,180 24375 [ main] INFO ome.formats.OMEROMetadataStoreClient - OS Version: 3.13.0-88-generic
2016-09-14 13:54:03,597 24792 [ main] INFO ome.formats.importer.ImportConfig - Using import target: Screen:758
2016-09-14 13:54:03,619 24814 [ main] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2016-09-14 13:54:04,194 25389 [ main] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2016-09-14 13:54:04,209 25404 [ main] INFO ts.importer.transfers.UploadFileTransfer - Transferring /data/share/ScanR/Zhi/FDAdrugs_Zhi_MCF7_DAPI_002/experiment_descriptor.xml...
2016-09-14 13:54:04,229 25424 [ main] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /data/share/ScanR/Zhi/FDAdrugs_Zhi_MCF7_DAPI_002/experiment_descriptor.xml
2016-09-14 13:54:04,264 25459 [ main] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /data/share/ScanR/Zhi/FDAdrugs_Zhi_MCF7_DAPI_002/experiment_descriptor.xml
2016-09-14 13:54:04,272 25467 [ main] INFO ts.importer.transfers.UploadFileTransfer - Transferring /data/share/ScanR/Zhi/FDAdrugs_Zhi_MCF7_DAPI_002/AcquisitionLog.dat...
2016-09-14 13:54:04,288 25483 [ main] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /data/share/ScanR/Zhi/FDAdrugs_Zhi_MCF7_DAPI_002/AcquisitionLog.dat
2016-09-14 13:54:04,307 25502 [ main] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /data/share/ScanR/Zhi/FDAdrugs_Zhi_MCF7_DAPI_002/AcquisitionLog.dat
2016-09-14 13:54:04,314 25509 [ main] INFO ts.importer.transfers.UploadFileTransfer - Transferring /data/share/ScanR/Zhi/FDAdrugs_Zhi_MCF7_DAPI_002/experiment_descriptor.dat...
2016-09-14 13:54:04,328 25523 [ main] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /data/share/ScanR/Zhi/FDAdrugs_Zhi_MCF7_DAPI_002/experiment_descriptor.dat
2016-09-14 13:54:04,406 25601 [ main] INFO ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /data/share/ScanR/Zhi/FDAdrugs_Zhi_MCF7_DAPI_002/experiment_descriptor.dat
2016-09-14 13:54:04,470 25665 [ main] INFO ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2016-09-14 13:54:04,559 25754 [ main] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 1994201
2016-09-14 13:54:04,618 25813 [l.Client-1] ERROR ome.formats.importer.cli.ErrorHandler - INTERNAL_EXCEPTION: /data/share/ScanR/Zhi/FDAdrugs_Zhi_MCF7_DAPI_002/experiment_descriptor.xml
java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: error-on-init
Parameters: {message=, stacktrace=java.lang.NullPointerException
}

at ome.formats.importer.ImportLibrary$ImportCallback.onFinished(ImportLibrary.java:666)
at omero.cmd.CmdCallbackI.finished(CmdCallbackI.java:334)
at omero.cmd._CmdCallbackDisp.___finished(_CmdCallbackDisp.java:118)
at omero.cmd._CmdCallbackDisp.__dispatch(_CmdCallbackDisp.java:145)
at IceInternal.Incoming.invoke(Incoming.java:222)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2482)
at Ice.ConnectionI.dispatch(ConnectionI.java:1258)
at Ice.ConnectionI.message(ConnectionI.java:1213)
at IceInternal.ThreadPool.run(ThreadPool.java:321)
at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:693)
at java.lang.Thread.run(Thread.java:745)

java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: error-on-init
Parameters: {message=, stacktrace=java.lang.NullPointerException
}

at ome.formats.importer.ImportLibrary$ImportCallback.onFinished(ImportLibrary.java:666) ~[blitz.jar:na]
at omero.cmd.CmdCallbackI.finished(CmdCallbackI.java:334) [blitz.jar:na]
at omero.cmd._CmdCallbackDisp.___finished(_CmdCallbackDisp.java:118) [blitz.jar:na]
.... etc ....


Thanks,
- Damir
dsudar
 
Posts: 235
Joined: Mon May 14, 2012 8:43 pm
Location: Berkeley, CA, USA

Re: new Olympus Scan^R filesets error on import

Postby dgault » Mon Sep 19, 2016 3:44 pm

Hi Damir,

Firstly thank you for providing the additional sample and my apologies for the slow response as I was on holiday last week.

Having tested the latest file it is a very similar issue to the previous one in that there is no suitable tag on the tif files that allow us to associate them with part of this fileset.

On our side we will have to further investigate the various differences between these software types in order to decide how to best handle them. The sample files provided here will certainly be of great use for that.

David
User avatar
dgault
Team Member
 
Posts: 208
Joined: Fri Aug 14, 2015 2:56 pm

Re: new Olympus Scan^R filesets error on import

Postby dsudar » Mon Oct 10, 2016 5:57 pm

Hi David,

Thanks for keeping this on your todo list. I meanwhile asked the Olympus ScanR folks why the format was changed, whether it's possible to put back the 305 IFD tag, and any advice how to resolve this issue. Here's the reply I received and I hope it may help in devising a solution:
since Version 2.6 we write native OME-Tiffs, there is no need for an
"Importer" anymore.

For this we write a "metadata.ome.xml" file into the data folder. Within
each acquired *.tiff there is a Link to this file, its in the tiff-tag
"305". In older versions only "National Instruments IMAQ" was writtten
there

One of our developers tested it during implementation with OMERO - however,
not your scanR-Import of course, but the compatibility of the saved
OME-Tiff files.

The solution is probably to remove a command you are using to tell OMERO
to open a scanR-scan(?) That wouldn't be necessary anymore.

I hope this information is helpful, but please don't hesitate to contact us
again in case you have further questions or the suggestion does not solve
the problem.


Since my users are stuck at this point, I'm looking for a temporary fix before the real solution comes out. Can you recommend a way to work around the issue? I'm thinking of something like:
1) import the data directory as a Dataset with some set of flags (help, please?) to create multi-channel Images from the 4 individual channel TIFFs
2) run the Dataset_To_Plate.py script to create a plate from the Dataset (but this will not work as is since Dataset_To_Plate doesn't deal with multiple fields).

Thanks,
- Damir
dsudar
 
Posts: 235
Joined: Mon May 14, 2012 8:43 pm
Location: Berkeley, CA, USA

Re: new Olympus Scan^R filesets error on import

Postby dsudar » Mon Oct 10, 2016 8:36 pm

One quick follow-up on my progress with a work-around:

I found that I can use the tiffset utility to insert a 305 tag with value "National Instruments IMAQ" into each of the TIFF files with simple one-line bash script:
Code: Select all
for fn in *.tif; do echo "working on $fn"; tiffset -s 305 "National Instruments IMAQ" "$fn"; done


Then the OMERO Importer can properly identify which TIFF files belong to the plate and the import proceeds correctly.

One additional small and easy to fix glitch: the new Olympus ScanR format now includes a file called metadata.ome.xml that it drops into the data directory. While that file itself appears like a good thing and has some valuable metadata info, it is not expected to be there and is not interpreted correctly by the importer and instead causes a ghost set of images to appear in the "orphaned images" container if it's left in the data directory. So for now one should remove that file before starting the import (and maybe put it back after the import has completed).

Cheers,
- Damir
dsudar
 
Posts: 235
Joined: Mon May 14, 2012 8:43 pm
Location: Berkeley, CA, USA

Re: new Olympus Scan^R filesets error on import

Postby sbesson » Tue Oct 11, 2016 5:03 pm

Hi Damir,

David is on leave this week so I will follow up as best as I can.

Thanks for all the information and big thumbs up for the workaround. If the tag 305 is already defined, overriding it with the expected string from the ScanR reader should be sufficient to have TIFF files listed in the reader used files.

On the `metadata.ome.xml` issue, this file is indeed not included in the ScanR fileset, flagged as an separate import candidate and imported separately as a ghost multi-image fileset (see below for details). Removing the file certainly works. Alternatively, if using the command-line import, I would expect the advanced --exclude option to work as follows:

Code: Select all
bin/omero import /path/to/ScanR --exclude=metadata.ome.xml


Generally, very glad to hear that Olympus ScanR people are evolving their stored data to use open formats. I understand the attempt is to store it as a multi-file OME-TIFF with a companion file containing the metadata.
Below are a couple of thoughts from our side on the current status/limitations based on the sample fileset you uploaded:

  • the OME-XML in the OME-XML file and the OME-TIFF headers are all fully valid according to the specification
  • although syntactically valid, the XML header only defines a list of images but contains no information about the SPW structure
  • assuming the intention is to create a companion file as every TIFF file refers to this file using the BinaryOnly, the companion extension must be `*.companion.ome` as per the specification.

At the moment,`metadata.ome.xml` is effectively treated and imported as an OME-XML file defining multiple images with no SPW structure leading to your ghost images. If this file was renamed `metadata.companion.ome` then I would expect it would be considered as an OME-TIFF fileset, that it should be imported alongside the corresponding TIFF files and that each imported image would contain the real pixel data. However, it would still lack the SPW structure as mentioned in my comment above.

Trying to move forward, please let us know if generating new OME-TIFF samples describing the expected structure would help. In all cases, if the Olympus ScanR folks want to contact us to get more information engage developer discussion, please feel free to redirect them to the public community resources.

Best,
Sebastien
User avatar
sbesson
Team Member
 
Posts: 421
Joined: Tue Feb 28, 2012 7:20 pm

PreviousNext

Return to User Discussion [Legacy]

Who is online

Users browsing this forum: No registered users and 0 guests