We're Hiring!

Whole dataset fails on import

General user discussion about using the OMERO platform to its fullest. Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

Whole dataset fails on import

Postby ratamero » Wed May 08, 2019 10:00 am

Hi there,

I'm still not sure whether this is an OMERO or Bioformats issue - I suspect the former. One of our users' datasets refuses to be imported into OMERO - every single .dv file in that specific acquisition fails. Both myself and the user managed to open those files in Fiji, which makes me think it's not a Bioformats issue. Whole error log for one of the files follow below; I'd be more than happy to submit one of the files for QA, but they're pretty big (~10GB each).

Code: Select all
2019-05-08 08:08:09,811 182        [      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.4.9-ice36-b101
2019-05-08 08:08:09,821 192        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 5.9.2 revision: 9fc607f85b8900be786813296f1eee75cc1ed883 date: 31 August 2018
2019-05-08 08:08:09,855 226        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2019-05-08 08:08:10,118 489        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2019-05-08 08:08:10,523 894        [      main] INFO      ome.formats.importer.ImportCandidates - 1 file(s) parsed into 1 group(s) with 1 call(s) to setId in 386ms. (405ms total) [0 unknowns]
2019-05-08 08:08:11,061 1432       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to camdu.warwick.ac.uk:4064
2019-05-08 08:08:11,362 1733       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2019-05-08 08:08:11,579 1950       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.4.9
2019-05-08 08:08:11,579 1950       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.4.9-ice36-b101
2019-05-08 08:08:11,579 1950       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 1.8.0_191
2019-05-08 08:08:11,579 1950       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2019-05-08 08:08:11,579 1950       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2019-05-08 08:08:11,580 1951       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 4.15.0-47-generic
2019-05-08 08:08:11,657 2028       [      main] INFO          ome.formats.importer.ImportConfig - Using import target: regex:^.*DV2/(?<Container1>.*?)
2019-05-08 08:08:11,700 2071       [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2019-05-08 08:08:12,163 2534       [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
2019-05-08 08:08:12,180 2551       [3-thread-1] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /home/erick/DV2/190425_tubulinanddna_mc150_1_5_mc179_6_10_mc180_10_15/190424_tubulinanddna_mc150_1_5_mc179_6_10_mc180_11_15_15_R3D.dv...
2019-05-08 08:08:12,199 2570       [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/erick/DV2/190425_tubulinanddna_mc150_1_5_mc179_6_10_mc180_10_15/190424_tubulinanddna_mc150_1_5_mc179_6_10_mc180_11_15_15_R3D.dv
2019-05-08 08:15:02,833 413204     [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/erick/DV2/190425_tubulinanddna_mc150_1_5_mc179_6_10_mc180_10_15/190424_tubulinanddna_mc150_1_5_mc179_6_10_mc180_11_15_15_R3D.dv
2019-05-08 08:15:02,850 413221     [3-thread-1] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /home/erick/DV2/190425_tubulinanddna_mc150_1_5_mc179_6_10_mc180_10_15/190424_tubulinanddna_mc150_1_5_mc179_6_10_mc180_11_15_15_R3D.dv.log...
2019-05-08 08:15:02,870 413241     [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /home/erick/DV2/190425_tubulinanddna_mc150_1_5_mc179_6_10_mc180_10_15/190424_tubulinanddna_mc150_1_5_mc179_6_10_mc180_11_15_15_R3D.dv.log
2019-05-08 08:15:03,252 413623     [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/erick/DV2/190425_tubulinanddna_mc150_1_5_mc179_6_10_mc180_10_15/190424_tubulinanddna_mc150_1_5_mc179_6_10_mc180_11_15_15_R3D.dv.log
2019-05-08 08:15:03,335 413706     [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
2019-05-08 08:15:03,392 413763     [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 684133
2019-05-08 08:15:11,017 421388     [l.Client-2] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5  Logfile: 684133
2019-05-08 08:19:28,168 678539     [l.Client-0] ERROR     ome.formats.importer.cli.ErrorHandler - INTERNAL_EXCEPTION: /home/erick/DV2/190425_tubulinanddna_mc150_1_5_mc179_6_10_mc180_10_15/190424_tubulinanddna_mc150_1_5_mc179_6_10_mc180_11_15_15_R3D.dv
java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: import-request-failure
Parameters: {stacktrace=java.lang.IllegalArgumentException
        at java.nio.Buffer.position(Buffer.java:244)
        at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:632)
        at loci.common.NIOFileHandle.skipBytes(NIOFileHandle.java:478)
        at loci.common.RandomAccessInputStream.skipBytes(RandomAccessInputStream.java:670)
        at loci.formats.FormatReader.readPlane(FormatReader.java:535)
        at loci.formats.FormatReader.readPlane(FormatReader.java:513)
        at loci.formats.in.DeltavisionReader.openBytes(DeltavisionReader.java:219)
        at loci.formats.ImageReader.openBytes(ImageReader.java:460)
        at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:156)
        at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:227)
        at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:348)
        at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:348)
        at loci.formats.MinMaxCalculator.openBytes(MinMaxCalculator.java:269)
        at ome.services.blitz.repo.ManagedImportRequestI.parseDataByPlane(ManagedImportRequestI.java:871)
        at ome.services.blitz.repo.ManagedImportRequestI.parseData(ManagedImportRequestI.java:802)
        at ome.services.blitz.repo.ManagedImportRequestI.pixelData(ManagedImportRequestI.java:675)
        at ome.services.blitz.repo.ManagedImportRequestI.step(ManagedImportRequestI.java:524)
        at omero.cmd.HandleI.steps(HandleI.java:448)
        at omero.cmd.HandleI$RunSteps.innerWork(HandleI.java:509)
        at omero.cmd.HandleI$2.doWork(HandleI.java:383)
        at omero.cmd.HandleI$2.doWork(HandleI.java:380)
        at sun.reflect.GeneratedMethodAccessor309.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:573)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        at com.sun.proxy.$Proxy72.doWork(Unknown Source)
        at ome.services.util.Executor$Impl.execute(Executor.java:452)
        at omero.cmd.HandleI.run(HandleI.java:379)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at ome.services.util.Executor$Impl$1.call(Executor.java:493)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
, message=}
<pre>
        at ome.formats.importer.ImportLibrary$ImportCallback.onFinished(ImportLibrary.java:756) ~[blitz.jar:na]
        at omero.cmd.CmdCallbackI.finished(CmdCallbackI.java:334) [blitz.jar:na]
        at omero.cmd._CmdCallbackDisp.___finished(_CmdCallbackDisp.java:118) [blitz.jar:na]
        at omero.cmd._CmdCallbackDisp.__dispatch(_CmdCallbackDisp.java:145) [blitz.jar:na]
        at IceInternal.Incoming.invoke(Incoming.java:221) [ice.jar:na]
        at Ice.ConnectionI.invokeAll(ConnectionI.java:2536) [ice.jar:na]
        at Ice.ConnectionI.dispatch(ConnectionI.java:1145) [ice.jar:na]
        at Ice.ConnectionI.message(ConnectionI.java:1056) [ice.jar:na]
        at IceInternal.ThreadPool.run(ThreadPool.java:395) [ice.jar:na]
        at IceInternal.ThreadPool.access$300(ThreadPool.java:12) [ice.jar:na]
        at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832) [ice.jar:na]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
2019-05-08 08:19:28,171 678542     [2-thread-1] ERROR        ome.formats.importer.ImportLibrary - Error on import
java.lang.Exception: Import failure
        at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:651) ~[blitz.jar:na]
        at ome.formats.importer.ImportLibrary$1.call(ImportLibrary.java:306) [blitz.jar:na]
        at ome.formats.importer.ImportLibrary$1.call(ImportLibrary.java:286) [blitz.jar:na]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_191]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_191]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
2019-05-08 08:19:28,171 678542     [2-thread-1] INFO         ome.formats.importer.ImportLibrary - Exiting on error
ratamero
 
Posts: 46
Joined: Mon Jan 29, 2018 5:55 pm

Re: Whole dataset fails on import

Postby ratamero » Thu May 09, 2019 11:16 am

Files uploaded via FTP. Thanks again!
ratamero
 
Posts: 46
Joined: Mon Jan 29, 2018 5:55 pm

Re: Whole dataset fails on import

Postby jmoore » Thu May 09, 2019 4:31 pm

Got 'em. Uploaded to erick/2019-05-09/
Thanks.
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Whole dataset fails on import

Postby ratamero » Tue Jun 04, 2019 8:34 am

I just thought it would be a good idea to bump this topic in case it falls through the cracks during forum migration :D
ratamero
 
Posts: 46
Joined: Mon Jan 29, 2018 5:55 pm

Re: Whole dataset fails on import

Postby mtbc » Mon Jun 10, 2019 9:53 am

Thanks, Erick, I don't see what we did after we got your data so I've created https://github.com/openmicroscopy/biofo ... ssues/3388 to be sure we track this problem. Sorry you ran into it!

Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Re: Whole dataset fails on import

Postby ratamero » Mon Jun 10, 2019 4:09 pm

No worries, thanks a lot!
ratamero
 
Posts: 46
Joined: Mon Jan 29, 2018 5:55 pm


Return to User Discussion

Who is online

Users browsing this forum: No registered users and 1 guest