LOCI tools issues in FiJi with Imaris files

[ehrenfeu]

Dear all,

I'm experiencing issues using Bio-Formats to read Imaris files (5.5, HDF5-based) in FiJi on Windows 64bit. While trying to reproduce my errors I probably discovered some more bugs unrelated to the ims file format as well. Here's what I did:

First of all, I downloaded a fresh ("vanilla", dating back to 2011-03-07) copy of FiJi from fiji.sc for Win64 bundled with JRE to make sure I have a clean setup. Next, I deleted my IJ_prefs.txt to exclude this as a possible source of problems. Then I tried all kinds of combinations of FiJi and loci_tools.jar to open an Imaris file (for my tests I created one using Imaris 7.5.0 having one channel and 5x5x5 voxels, but I also tried on other ims files with identical results).

For the FiJi vanilla with recent LOCI tools combinations I used the integrated LOCI updater, so except for the loci_tools.jar this was still the plain and fresh FiJi environment as unpacked from the ZIP. For the FiJi versions marked as "latest" all updates available via the FiJi updater were installed with no extra update sites configured but the two default ones (ImageJ + FiJi). I've condensed my observations here:

Code: Select all

| FiJi version          |  LOCI version             |  read imaris file     | notes
----------------------------------------------------------------------------------------
| vanilla (2011-03-07)  |  vanilla (2011-02-16)     |  ok                   |
| vanilla (2011-03-07)  |  trunk (2012-01-02)       |  ok                   |
| vanilla (2011-03-07)  |  stable (2012-08-25)      |  ok                   |  [1]
| vanilla (2011-03-07)  |  daily (2012-09-04)       |  ok                   |  [1]
| latest (2012-09-04)   |  vanilla (2011-02-16)     |  ERROR                |
| latest (2012-09-04)   |  trunk (2012-01-02)       |  ERROR                |
| latest (2012-09-04)   |  stable (2012-08-25)      |  ERROR                |  [1]
| latest (2012-09-04)   |  daily (2012-09-04)       |  ERROR                |  [1]
| latest (2012-09-04)   |  included (4.4-SNAPSHOT)  |  no file open dialog  |  [2]


Most surprising to me was the behavior of the last combination, not even the file open dialog was shown there (and nothing strange is shown in ImageJ's debug mode). BTW, this was not true for my local FiJi installation where the LOCI repository has been added to the FiJi update sites, here the file open window is displayed (imaris files still can't be opened though).

The version marked with "ERROR" above display the following message after a file has been chosen and the settings dialog was presented (line numbers vary across different versions):

Code: Select all

java.lang.IllegalArgumentException: Invalid C size: 0
   at loci.formats.FormatTools.getZCTCoords(FormatTools.java:408)
   at loci.formats.FormatTools.getZCTCoords(FormatTools.java:369)
   at loci.formats.FormatReader.getZCTCoords(FormatReader.java:980)
   at loci.formats.in.ImarisHDFReader.getImageData(ImarisHDFReader.java:330)
   at loci.formats.in.ImarisHDFReader.initFile(ImarisHDFReader.java:225)
   at loci.formats.FormatReader.setId(FormatReader.java:1178)
   at loci.plugins.in.ImportProcess.initializeFile(ImportProcess.java:475)
   at loci.plugins.in.ImportProcess.execute(ImportProcess.java:138)
   at loci.plugins.in.Importer.showDialogs(Importer.java:125)
   at loci.plugins.in.Importer.run(Importer.java:77)
   at loci.plugins.LociImporter.run(LociImporter.java:77)
   at ij.IJ.runUserPlugIn(IJ.java:185)
   at ij.IJ.runPlugIn(IJ.java:152)
   at ij.IJ.runPlugIn(IJ.java:141)
   at loci.plugins.shortcut.ShortcutPanel$1.run(ShortcutPanel.java:189)


NOTES:

[1]: calling the LOCI updater from this version yields an error dialog showing: "An error occurred while downloading the LOCI plugins."

[2]: calling the LOCI updater from this version results in an error message displayed within the log window:

Code: Select all

java.lang.ExceptionInInitializerError
   at loci.plugins.Updater.install(Updater.java:98)
   at loci.plugins.Updater.run(Updater.java:87)
   at ij.IJ.runUserPlugIn(IJ.java:185)
   at ij.IJ.runPlugIn(IJ.java:152)
   at ij.IJ.runPlugIn(IJ.java:141)
   at loci.plugins.shortcut.ShortcutPanel$1.run(ShortcutPanel.java:192)
Caused by: java.lang.IllegalStateException: org.slf4j.LoggerFactory could not be successfully initialized. See also http://www.slf4j.org/codes.html#unsuccessfulInit
   at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:282)
   at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:248)
   at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:261)
   at loci.formats.UpgradeChecker.<clinit>(UpgradeChecker.java:126)
   ... 6 more


My Imaris test file is attached to this post. Please note it's NOT a ZIP file, just remove the ".zip" suffix from the filename (had to add it due to the forum software not accepting ".ims" files... very "safe" )

Any help is greatly appreciated!

Niko

[mlinkert]

Hi Niko,

Thank you for the detailed report.

We have already fixed this error:

Code: Select all

java.lang.IllegalArgumentException: Invalid C size: 0
   at loci.formats.FormatTools.getZCTCoords(FormatTools.java:408)
   at loci.formats.FormatTools.getZCTCoords(FormatTools.java:369)
   at loci.formats.FormatReader.getZCTCoords(FormatReader.java:980)
   at loci.formats.in.ImarisHDFReader.getImageData(ImarisHDFReader.java:330)
   at loci.formats.in.ImarisHDFReader.initFile(ImarisHDFReader.java:225)
   at loci.formats.FormatReader.setId(FormatReader.java:1178)
   at loci.plugins.in.ImportProcess.initializeFile(ImportProcess.java:475)
   at loci.plugins.in.ImportProcess.execute(ImportProcess.java:138)
   at loci.plugins.in.Importer.showDialogs(Importer.java:125)
   at loci.plugins.in.Importer.run(Importer.java:77)
   at loci.plugins.LociImporter.run(LociImporter.java:77)
   at ij.IJ.runUserPlugIn(IJ.java:185)
   at ij.IJ.runPlugIn(IJ.java:152)
   at ij.IJ.runPlugIn(IJ.java:141)
   at loci.plugins.shortcut.ShortcutPanel$1.run(ShortcutPanel.java:189)


...and so the upcoming 4.4.4 release should allow the file to open.

With respect to the other errors and odd behavior, I believe much of this is down to the fact that recent updates to Fiji have changed the way that Bio-Formats is packaged in Fiji. Among other things, this means that "Plugins > LOCI > Update LOCI Plugins" broke, and simply dropping in a replacement loci_tools.jar will no longer correctly update Bio-Formats.

The simplest solution is to wait a week or two until we have released Bio-Formats 4.4.4, which I hope very much will be put in the Fiji updater. Then you should be able to just update Fiji through "Help > Update Fiji" and the file should be readable (and after that, "Plugins > LOCI > Update LOCI Plugins" should work again).

-Melissa

[ehrenfeu]

Hi Melissa,

thanks for your quick response. Glad to hear the Imaris bug is already fixed. Is there a nightly build available somewhere, since I can't just drop in a new loci_tools.jar as you said...?

There's one more thing I've noticed, no clue if this is also related to FiJi's packaging: the "About" dialog doesn't show the revision, build date or release string any more - apparently the substitution responsible for filling in the right values doesn't happen, instead the placeholders (like "@date@") are still in there.

~Niko

[mlinkert]

Hi Niko,

thanks for your quick response. Glad to hear the Imaris bug is already fixed. Is there a nightly build available somewhere, since I can't just drop in a new loci_tools.jar as you said...?

If you want to try out the latest build, you will need to download bio-formats.jar, scifio.jar, loci-common.jar, ome-xml.jar, loci_plugins.jar, metakit.jar, jai_imageio.jar, mdbtools_java.jar, and poi-loci.jar from the "Bio-Formats component builds" section of:

http://loci.wisc.edu/bio-formats/downloads

loci_plugins.jar goes in your Fiji plugins folder; everything else goes in Fiji's 'jars' folder. Note that there will be similarly-named files there already, so make sure to overwrite the existing files.

There's one more thing I've noticed, no clue if this is also related to FiJi's packaging: the "About" dialog doesn't show the revision, build date or release string any more - apparently the substitution responsible for filling in the right values doesn't happen, instead the placeholders (like "@date@") are still in there.

Yes, this is related. The .jar files used by Fiji now are built using a different build system (Maven), which does not bother to replace the "@date", "@version", etc. with the actual date and version. This is noted in this ticket:

http://trac.openmicroscopy.org.uk/ome/ticket/8797

-Melissa

[ehrenfeu]

Dear Melissa,

I'm still experiencing almost the identical error (differs basically just in line numbers) with an up-to-date FiJi using the LOCI plugins v4.4.4 (built 25 September):

Code: Select all

java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at ij.Command.runPlugIn(Command.java:146)
   at ij.Command.runCommand(Command.java:95)
   at ij.Executer.run(Executer.java:64)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgumentException: Invalid C size: 0
   at loci.formats.FormatTools.getZCTCoords(FormatTools.java:408)
   at loci.formats.FormatTools.getZCTCoords(FormatTools.java:369)
   at loci.formats.FormatReader.getZCTCoords(FormatReader.java:980)
   at loci.formats.in.ImarisHDFReader.getImageData(ImarisHDFReader.java:420)
   at loci.formats.in.ImarisHDFReader.initFile(ImarisHDFReader.java:288)
   at loci.formats.FormatReader.setId(FormatReader.java:1178)
   at loci.plugins.in.ImportProcess.initializeFile(ImportProcess.java:482)
   at loci.plugins.in.ImportProcess.execute(ImportProcess.java:146)
   at loci.plugins.in.Importer.showDialogs(Importer.java:141)
   at loci.plugins.in.Importer.run(Importer.java:79)
   at loci.plugins.LociImporter.run(LociImporter.java:81)
   at ij.IJ.runUserPlugIn(IJ.java:185)
   at ij.IJ.runPlugIn(IJ.java:152)
   ... 8 more


Is this a regression or did the fix not go into 4.4.4?

Thanks for your help!
Niko

[mlinkert]

Hi Niko,

Unfortunately, I can't duplicate this problem with 4.4.4 and your imaris-testfile_5x5x5.ims file (md5sum = 5a09e552a6d165b4adc8da77f192b8f9).

One possibility is that there are some old .jar files in your Fiji installation which are conflicting with the 4.4.4 .jars. This has been a fairly common problem over the last couple of months, so I would suggest going into the Fiji updater, clicking "Advanced mode", and then choosing "View local-only files". If all is well, there should be no loci_tools.jar there, nor anything that contains "4.4-SNAPSHOT". If you do see anything like that, then please uninstall it and restart Fiji.

If resolving any updater issues does not solve this problem, then could you please indicate which options were checked in the "Bio-Formats Import Options" window?

-Melissa

[ehrenfeu]

Ok, I will investigate this tomorrow.

~Niko

[ehrenfeu]

Unfortunately, I can't duplicate this problem with 4.4.4 and your imaris-testfile_5x5x5.ims file (md5sum = 5a09e552a6d165b4adc8da77f192b8f9).

Hum. That's strange. I double-checked now again, md5sum is identical but I still get the error.

One possibility is that there are some old .jar files in your Fiji installation which are conflicting with the 4.4.4 .jars. This has been a fairly common problem over the last couple of months, so I would suggest going into the Fiji updater, clicking "Advanced mode", and then choosing "View local-only files". If all is well, there should be no loci_tools.jar there, nor anything that contains "4.4-SNAPSHOT". If you do see anything like that, then please uninstall it and restart Fiji.

I now did a full test on win7/x64 starting with removing all my local configuration (IJ_Prefs.txt from %USERPROFILE%), then extracting a fresh vanilla copy of FiJi (on my Desktop to make sure I have the appropriate file system permissions), running all the updates until no more updates are available. So ImageJ is on 1.47d, the LOCI plugins on 4.4.4. No strange local files exist.

Please see the attached screenshot for details, in case I'm missing something...

If resolving any updater issues does not solve this problem, then could you please indicate which options were checked in the "Bio-Formats Import Options" window?

There was nothing ticked except for the "autoscale" option. Unchecking it doesn't make a difference.

When trying to use the "Update LOCI Plugins" mechanism and selecting "Daily" (or one of the other choices), it starts downloading some .jar files but then stops with the following error:

Code: Select all

java.lang.NullPointerException
   at java.io.File.<init>(File.java:222)
   at loci.formats.UpgradeChecker.install(UpgradeChecker.java:341)
   at loci.plugins.Updater.install(Updater.java:118)
   at loci.plugins.Updater.run(Updater.java:94)
   at ij.IJ.runUserPlugIn(IJ.java:185)
   at ij.IJ.runPlugIn(IJ.java:152)
   at ij.IJ.runPlugIn(IJ.java:141)
   at loci.plugins.shortcut.ShortcutPanel$1.run(ShortcutPanel.java:192)


Any hints?

~Niko

[ehrenfeu]

Update: FiJi shows the same behaviour on my Linux (Ubuntu 11.10, 64 bit) - the imaris reader error as well as the update-loci-plugins...

(again, started from scratch with a plain new fiji and no local ij_prefs)

[mlinkert]

Hi Niko,

Thank you very much for the additional information. I can indeed now duplicate both problems.

The Imaris file reading problem is a bit complex. Bio-Formats relies upon a separate library (netCDF) to read HDF-based files. Bio-Formats is written against the 4.0.03 version of netCDF, and if you use loci_tools.jar or download the separate .jar files from http://loci.wisc.edu/bio-formats/bio-fo ... va-library then that is the version that you will get. This is important because netCDF is known to have various small inconsistencies between versions, which can affect file reading in certain cases. Unfortunately, Fiji includes netCDF 4.0.41 and not 4.0.03; the differences between those two versions turn out to be the cause of the first error that you are seeing. Overwriting Fiji's jars/netcdf-java-4.0.41.jar with the netcdf-4.0.jar file from the above page should solve the problem. The Fiji updater will complain about that, but it should let you at least open your files.

The "Plugins > LOCI > Update LOCI Plugins" issue is thankfully much simpler, and is ticketed here:

http://trac.openmicroscopy.org.uk/ome/ticket/9794

-Melissa