We're Hiring!

Issue with bfconvert using flags

Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats
Please note:
Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats

If you are having trouble with image files, there is information about reporting bugs in the Bio-Formats documentation. Please send us the data and let us know what version of Bio-Formats you are using. For issues with your code, please provide a link to a public repository, ideally GitHub.

Issue with bfconvert using flags

Postby tim0 » Wed Feb 08, 2017 1:11 am

This issue applies to:
bfconvert
Version: 5.3.3
Build date: 1 February 2017
VCS revision: 5.3.3

When using bfconvert to convert .ND2 to .tiff (also .ome.tiff) there is a reproducible error when using the flags -series, -channel, -timepoint, and -z. The error is always "[Fatal Error] :1:72: Character reference "&#55663" is an invalid XML character." more details below. This error does not occur with bfconvert version 5.1.10. The ND2 files are created by a Nikon DS-Qi2 camera. The files are converted correctly (not strictly true, but that is a separate issue) when the flags are omitted.

This error is reproducible with any .nd2 produced on this camera (I have not tried other cameras), but I have made a blank .nd2 including multiple series, channels, timepoints, and z for testing purposes which I am happy to provide to anyone (it is 357 MB when compressed in .7z format or 1.54 GB in .nd2 format).

example of issue:
C:\Users\tbates3\Desktop\Temp\bftools-new> bfconvert C:\Users\tbates3\Desktop\Testing\supernd2.nd2 C:\Users\tbates3\Desktop\TiffOutput\test3.tiff
-This works and produces a multipage tiff file

C:\Users\tbates3\Desktop\Temp\bftools-new> bfconvert -debug -series 0 C:\Users\tbates3\Desktop\Testing\supernd2.nd2 C:\Users\tbates3\Desktop\TiffOutput\test3.tiff
No update needed
C:\Users\tbates3\Desktop\Testing\supernd2.nd2
Could not find loci.formats.in.SlideBook6Reader
java.lang.ClassNotFoundException: loci.formats.in.SlideBook6Reader
at java.net.URLClassLoader.findClass(Unknown Source) ~[na:1.8.0_121]
at java.lang.ClassLoader.loadClass(Unknown Source) ~[na:1.8.0_121]
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) ~[na:1.8.0
_121]
at java.lang.ClassLoader.loadClass(Unknown Source) ~[na:1.8.0_121]
at java.lang.Class.forName0(Native Method) ~[na:1.8.0_121]
at java.lang.Class.forName(Unknown Source) ~[na:1.8.0_121]
at loci.formats.ClassList.parseLine(ClassList.java:196) [bioformats_pack
age.jar:5.3.2]
at loci.formats.ClassList.parseFile(ClassList.java:258) [bioformats_pack
age.jar:5.3.2]
at loci.formats.ClassList.<init>(ClassList.java:138) [bioformats_package
.jar:5.3.2]
at loci.formats.ClassList.<init>(ClassList.java:122) [bioformats_package
.jar:5.3.2]
at loci.formats.ImageReader.getDefaultReaderClasses(ImageReader.java:79)
[bioformats_package.jar:5.3.2]
at loci.formats.ImageReader.<init>(ImageReader.java:116) [bioformats_pac
kage.jar:5.3.2]
at loci.formats.tools.ImageConverter.testConvert(ImageConverter.java:341
) [bioformats_package.jar:5.3.2]
at loci.formats.tools.ImageConverter.main(ImageConverter.java:880) [biof
ormats_package.jar:5.3.2]
processor is INTEL_64 os.arch is amd64
architecture is WINDOWS_64 os.name is windows 7
architecture is WINDOWS_64 os.name is windows 7
platform specific path is META-INF/lib/windows_64/
mappedLib is turbojpeg.dll
URL is jar:file:/C:/Users/tbates3/Desktop/Temp/bftools-new/bioformats_package.ja
r!/META-INF/lib/windows_64/turbojpeg.dll
URL path is file:/C:/Users/tbates3/Desktop/Temp/bftools-new/bioformats_package.j
ar!/META-INF/lib/windows_64/turbojpeg.dll
Extracting 'jar:file:/C:/Users/tbates3/Desktop/Temp/bftools-new/bioformats_packa
ge.jar!/META-INF/lib/windows_64/turbojpeg.dll' to 'C:\Users\tbates3\AppData\Loca
l\Temp\turbojpeg6688584493340154469.dll'
Loaded properties from: services.properties
Added interface interface loci.formats.services.POIService and implementation cl
ass loci.formats.services.POIServiceImpl
Added interface interface loci.formats.services.MDBService and implementation cl
ass loci.formats.services.MDBServiceImpl
Added interface interface loci.formats.services.JPEGTurboService and implementat
ion class loci.formats.services.JPEGTurboServiceImpl
Added interface interface loci.formats.services.JAIIIOService and implementation
class loci.formats.services.JAIIIOServiceImpl
Added interface interface loci.formats.services.WlzService and implementation cl
ass loci.formats.services.WlzServiceImpl
Added interface interface loci.formats.services.JHDFService and implementation c
lass loci.formats.services.JHDFServiceImpl
Added interface interface loci.formats.services.NetCDFService and implementation
class loci.formats.services.NetCDFServiceImpl
Added interface interface loci.formats.services.EXIFService and implementation c
lass loci.formats.services.EXIFServiceImpl
Added interface interface loci.formats.services.MetakitService and implementatio
n class loci.formats.services.MetakitServiceImpl
Added interface interface loci.formats.services.LuraWaveService and implementati
on class loci.formats.services.LuraWaveServiceImpl
Added interface interface loci.formats.services.OMEXMLService and implementation
class loci.formats.services.OMEXMLServiceImpl
Added interface interface loci.formats.services.JPEGXRService and implementation
class loci.formats.services.JPEGXRServiceImpl
Using mapped byte buffer? false
ND2Reader initializing C:\Users\tbates3\Desktop\Testing\supernd2.nd2
NativeND2Reader initializing C:\Users\tbates3\Desktop\Testing\supernd2.nd2
loci.formats.in.NativeND2Reader.initFile(C:\Users\tbates3\Desktop\Testing\supern
d2.nd2)
Searching for blocks
Parsing block 'ND2 FILE SIG' 0%
Parsing block 'ImageMetadat' 0%
Parsing block 'ImageCalibra' 0%
Parsing block 'ImageMetadat' 0%
Parsing block 'ImageCalibra' 0%
Parsing block 'ImageMetadat' 0%
Parsing block 'ImageAttribu' 0%
Parsing block 'ImageDataSeq' 0%
Parsing block 'ImageDataSeq' 1%
Parsing block 'ImageDataSeq' 2%
Parsing block 'ImageDataSeq' 3%
Parsing block 'ImageDataSeq' 5%
Parsing block 'ImageDataSeq' 6%
Parsing block 'ImageDataSeq' 7%
Parsing block 'ImageDataSeq' 8%
Parsing block 'ImageDataSeq' 10%
Parsing block 'ImageDataSeq' 11%
Parsing block 'ImageDataSeq' 12%
Parsing block 'ImageDataSeq' 13%
Parsing block 'ImageDataSeq' 15%
Parsing block 'ImageDataSeq' 16%
Parsing block 'ImageDataSeq' 17%
Parsing block 'ImageDataSeq' 18%
Parsing block 'ImageDataSeq' 20%
Parsing block 'ImageDataSeq' 21%
Parsing block 'ImageDataSeq' 22%
Parsing block 'ImageDataSeq' 23%
Parsing block 'ImageDataSeq' 25%
Parsing block 'ImageDataSeq' 26%
Parsing block 'ImageDataSeq' 27%
Parsing block 'ImageDataSeq' 28%
Parsing block 'ImageDataSeq' 30%
Parsing block 'ImageDataSeq' 31%
Parsing block 'ImageDataSeq' 32%
Parsing block 'ImageDataSeq' 33%
Parsing block 'ImageDataSeq' 35%
Parsing block 'ImageDataSeq' 36%
Parsing block 'ImageDataSeq' 37%
Parsing block 'ImageDataSeq' 38%
Parsing block 'ImageDataSeq' 40%
Parsing block 'ImageDataSeq' 41%
Parsing block 'ImageDataSeq' 42%
Parsing block 'ImageDataSeq' 43%
Parsing block 'ImageDataSeq' 45%
Parsing block 'ImageDataSeq' 46%
Parsing block 'ImageDataSeq' 47%
Parsing block 'ImageDataSeq' 48%
Parsing block 'ImageDataSeq' 49%
Parsing block 'ImageDataSeq' 51%
Parsing block 'ImageDataSeq' 52%
Parsing block 'ImageDataSeq' 53%
Parsing block 'ImageDataSeq' 54%
Parsing block 'ImageDataSeq' 56%
Parsing block 'ImageDataSeq' 57%
Parsing block 'ImageDataSeq' 58%
Parsing block 'ImageDataSeq' 59%
Parsing block 'ImageDataSeq' 61%
Parsing block 'ImageDataSeq' 62%
Parsing block 'ImageDataSeq' 63%
Parsing block 'ImageDataSeq' 64%
Parsing block 'ImageDataSeq' 66%
Parsing block 'ImageDataSeq' 67%
Parsing block 'ImageDataSeq' 68%
Parsing block 'ImageDataSeq' 69%
Parsing block 'ImageDataSeq' 71%
Parsing block 'ImageDataSeq' 72%
Parsing block 'ImageDataSeq' 73%
Parsing block 'ImageDataSeq' 74%
Parsing block 'ImageDataSeq' 76%
Parsing block 'ImageDataSeq' 77%
Parsing block 'ImageDataSeq' 78%
Parsing block 'ImageDataSeq' 79%
Parsing block 'ImageDataSeq' 81%
Parsing block 'ImageDataSeq' 82%
Parsing block 'ImageDataSeq' 83%
Parsing block 'ImageDataSeq' 84%
Parsing block 'ImageDataSeq' 86%
Parsing block 'ImageDataSeq' 87%
Parsing block 'ImageDataSeq' 88%
Parsing block 'ImageDataSeq' 89%
Parsing block 'ImageDataSeq' 91%
Parsing block 'ImageDataSeq' 92%
Parsing block 'ImageDataSeq' 93%
Parsing block 'ImageDataSeq' 94%
Parsing block 'ImageDataSeq' 96%
Parsing block 'ImageDataSeq' 97%
Parsing block 'ImageDataSeq' 98%
Parsing block 'ImageMetadat' 99%
Parsing block 'CustomData|R' 99%
Parsing block 'ImageEventsL' 99%
Parsing block 'ImageTextInf' 99%
Parsing block 'CustomDataVa' 99%
Parsing block 'CustomDataVa' 99%
Parsing block 'CustomDataVa' 99%
Parsing block 'CustomDataVa' 99%
Parsing block 'CustomDataVa' 99%
Parsing block 'CustomDataVa' 99%
Parsing block 'CustomData|C' 99%
Parsing block 'CustomData|P' 99%
Parsing block 'CustomData|P' 99%
Parsing block 'CustomData|X' 99%
Parsing block 'CustomData|Y' 99%
Parsing block 'CustomData|Z' 99%
Parsing block 'CustomData|T' 99%
Parsing block 'CustomData|A' 99%
Parsing block 'CustomData|A' 99%
Parsing block 'CustomData|A' 99%
Parsing block 'CustomDataVa' 99%
Parsing block 'ImageAttribu' 99%
Parsing block 'ND2 FILEMAP ' 99%
Could not parse XML
java.io.IOException: null
at loci.common.xml.XMLTools.parseXML(XMLTools.java:452) ~[bioformats_pac
kage.jar:5.3.2]
at loci.common.xml.XMLTools.parseXML(XMLTools.java:430) ~[bioformats_pac
kage.jar:5.3.2]
at loci.common.xml.XMLTools.parseXML(XMLTools.java:409) ~[bioformats_pac
kage.jar:5.3.2]
at loci.formats.in.NativeND2Reader.parseText(NativeND2Reader.java:2146)
[bioformats_package.jar:5.3.2]
at loci.formats.in.NativeND2Reader.initFile(NativeND2Reader.java:754) [b
ioformats_package.jar:5.3.2]
at loci.formats.FormatReader.setId(FormatReader.java:1399) [bioformats_p
ackage.jar:5.3.2]
at loci.formats.DelegateReader.setId(DelegateReader.java:291) [bioformat
s_package.jar:5.3.2]
at loci.formats.ImageReader.setId(ImageReader.java:839) [bioformats_pack
age.jar:5.3.2]
at loci.formats.tools.ImageConverter.testConvert(ImageConverter.java:380
) [bioformats_package.jar:5.3.2]
at loci.formats.tools.ImageConverter.main(ImageConverter.java:880) [biof
ormats_package.jar:5.3.2]
Caused by: org.xml.sax.SAXParseException: Content is not allowed in prolog.
at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Un
known Source) ~[bioformats_package.jar:5.3.2]
at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)
~[bioformats_package.jar:5.3.2]
at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) ~
[bioformats_package.jar:5.3.2]
at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source) ~
[bioformats_package.jar:5.3.2]
at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source) ~[
bioformats_package.jar:5.3.2]
at org.apache.xerces.impl.XMLDocumentScannerImpl$PrologDispatcher.dispat
ch(Unknown Source) ~[bioformats_package.jar:5.3.2]
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Un
known Source) ~[bioformats_package.jar:5.3.2]
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) ~[
bioformats_package.jar:5.3.2]
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) ~[
bioformats_package.jar:5.3.2]
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) ~[bioformat
s_package.jar:5.3.2]
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) ~[b
ioformats_package.jar:5.3.2]
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Sour
ce) ~[bioformats_package.jar:5.3.2]
at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source) ~[bioforma
ts_package.jar:5.3.2]
at javax.xml.parsers.SAXParser.parse(Unknown Source) ~[na:1.8.0_121]
at loci.common.xml.XMLTools.parseXML(XMLTools.java:444) ~[bioformats_pac
kage.jar:5.3.2]
... 9 common frames omitted
[Fatal Error] :1:72: Character reference "&#55663" is an invalid XML character.
Exception in thread "main" java.lang.RuntimeException: org.xml.sax.SAXParseExcep
tion; lineNumber: 1; columnNumber: 72; Character reference "&#55663" is an inval
id XML character.
at ome.xml.model.XMLAnnotation.asXMLElement(XMLAnnotation.java:265)
at ome.xml.model.XMLAnnotation.asXMLElement(XMLAnnotation.java:234)
at ome.xml.model.StructuredAnnotations.asXMLElement(StructuredAnnotation
s.java:679)
at ome.xml.model.StructuredAnnotations.asXMLElement(StructuredAnnotation
s.java:660)
at ome.xml.model.OME.asXMLElement(OME.java:896)
at ome.xml.model.OME.asXMLElement(OME.java:773)
at ome.xml.meta.AbstractOMEXMLMetadata.dumpXML(AbstractOMEXMLMetadata.ja
va:110)
at ome.xml.meta.OMEXMLMetadataImpl.dumpXML(OMEXMLMetadataImpl.java:105)
at loci.formats.services.OMEXMLServiceImpl.getOMEXML(OMEXMLServiceImpl.j
ava:465)
at loci.formats.tools.ImageConverter.testConvert(ImageConverter.java:414
)
at loci.formats.tools.ImageConverter.main(ImageConverter.java:880)
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 72; Chara
cter reference "&#55663" is an invalid XML character.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at ome.xml.model.XMLAnnotation.asXMLElement(XMLAnnotation.java:261)
... 10 more

Thank you,
Tim
tim0
 
Posts: 3
Joined: Wed Feb 01, 2017 6:27 pm

Re: Issue with bfconvert using flags

Postby bramalingam » Wed Feb 08, 2017 3:29 pm

Hi,

Thank you for reporting your issue.

We had recently fixed issues with our ImageConverter in this PR,
https://github.com/openmicroscopy/bioformats/pull/2737
The fix was included in the 5.3.3 release of Bio-Formats.

Testing with the files we have locally, we were unable to reproduce the issue that you have reported.

It would be great if you could submit one of your nd2 files to our QA server,
https://www.openmicroscopy.org/qa2/qa/upload/

This would help us troubleshoot the issue in better detail.

Best,
Balaji
User avatar
bramalingam
 
Posts: 70
Joined: Tue Jan 14, 2014 12:01 pm

Re: Issue with bfconvert using flags

Postby tim0 » Fri Feb 10, 2017 8:56 pm

I have uploaded a file called supernd2.zip which contains an nd2 with all of the available options in our software, NIS Elements AR (several of each timepoints, series, channels, z-series), but all images are of the inside of the camera shutter. This file gives the same results as my files containing actual data.

I also verified that my java is up to date with Version 8 build 121. My OS is Windows 7 Enterprise.

Cheers,
Tim
tim0
 
Posts: 3
Joined: Wed Feb 01, 2017 6:27 pm

Re: Issue with bfconvert using flags

Postby bramalingam » Mon Feb 13, 2017 5:48 pm

Hi,

Thank you for submitting your files.
We were able to reproduce the issue now with Bio-Foramts 5.3.3.

We will be filing a ticket on this front, please let us know if you would like to get added to the cc list on the ticket.

Best,
Balaji
User avatar
bramalingam
 
Posts: 70
Joined: Tue Jan 14, 2014 12:01 pm

Re: Issue with bfconvert using flags

Postby tim0 » Tue Mar 28, 2017 7:56 pm

Hello,

Just to let you know, as of 5.4.0 the error is now occurring every time regardless of the use of flags. 5.1.10 still works correctly though in all cases tested by me, so I will just keep using that for the time being.

Cheers,
Tim
tim0
 
Posts: 3
Joined: Wed Feb 01, 2017 6:27 pm

Re: Issue with bfconvert using flags

Postby dgault » Wed Mar 29, 2017 1:03 pm

Hi Tim,

Thanks for providing an update. I have added those details to the ticket (https://trac.openmicroscopy.org/ome/ticket/13322) and will increase the priority for further investigation.

David Gault
User avatar
dgault
Team Member
 
Posts: 208
Joined: Fri Aug 14, 2015 2:56 pm


Return to User Discussion [Legacy]

Who is online

Users browsing this forum: No registered users and 1 guest

cron