Open Microscopy Environment

The OME community
https://www.openmicroscopy.org/community/

image import python script

https://www.openmicroscopy.org/community/viewtopic.php?f=6&t=895

Page 2 of 2

Re: image import python script

PostPosted: Fri Nov 25, 2011 8:30 pm
by bhcho
I have 8M file after I zipped it.
Could you tell me how to send this to you? I don't think I can attach it here.

BK

Re: image import python script

PostPosted: Fri Nov 25, 2011 9:04 pm
by bhcho
Using the java option that Josh told me above, I still have the error when I import 400 images using the shell script.
here's the output message.

2011-11-25 15:56:24,953 133209 [ main] ERROR ome.formats.importer.cli.ErrorHandler - INTERNAL_EXCEPTION: /share/images/3T3/RandTag/for_BK/Mito073P20Red/16/100$
java.lang.RuntimeException: omero.InternalException
serverStackTrace = "ome.conditions.InternalException: Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded
"
serverExceptionClass = "ome.conditions.InternalException"
message = " Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded"
at ome.formats.OMEROMetadataStoreClient.populateMinMax(OMEROMetadataStoreClient.java:2429)
at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:647)
at ome.formats.importer.ImportLibrary.importCandidates(ImportLibrary.java:223)
at ome.formats.importer.cli.CommandLineImporter.start(CommandLineImporter.java:128)
at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:366)
Caused by: omero.InternalException
serverStackTrace = "ome.conditions.InternalException: Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded
"
serverExceptionClass = "ome.conditions.InternalException"
message = " Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded"
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at IceInternal.BasicStream$DynamicUserExceptionFactory.createAndThrow(BasicStream.java:2243)
at IceInternal.BasicStream.throwException(BasicStream.java:1632)
at IceInternal.Outgoing.throwUserException(Outgoing.java:442)
at omero.api._MetadataStoreDelM.populateMinMax(_MetadataStoreDelM.java:79)
at omero.api.MetadataStorePrxHelper.populateMinMax(MetadataStorePrxHelper.java:117)
at omero.api.MetadataStorePrxHelper.populateMinMax(MetadataStorePrxHelper.java:89)
at ome.formats.OMEROMetadataStoreClient.populateMinMax(OMEROMetadataStoreClient.java:2425)
... 4 more

java.lang.RuntimeException: omero.InternalException
serverStackTrace = "ome.conditions.InternalException: Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded
"
serverExceptionClass = "ome.conditions.InternalException"
message = " Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded"
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at IceInternal.BasicStream$DynamicUserExceptionFactory.createAndThrow(BasicStream.java:2243)
at IceInternal.BasicStream.throwException(BasicStream.java:1632)
at IceInternal.Outgoing.throwUserException(Outgoing.java:442)
at omero.api._MetadataStoreDelM.populateMinMax(_MetadataStoreDelM.java:79)
at omero.api.MetadataStorePrxHelper.populateMinMax(MetadataStorePrxHelper.java:117)
at omero.api.MetadataStorePrxHelper.populateMinMax(MetadataStorePrxHelper.java:89)
at ome.formats.OMEROMetadataStoreClient.populateMinMax(OMEROMetadataStoreClient.java:2425)
... 4 more

java.lang.RuntimeException: omero.InternalException
serverStackTrace = "ome.conditions.InternalException: Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded
"
serverExceptionClass = "ome.conditions.InternalException"
message = " Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded"
at ome.formats.OMEROMetadataStoreClient.populateMinMax(OMEROMetadataStoreClient.java:2429)
at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:647)
at ome.formats.importer.ImportLibrary.importCandidates(ImportLibrary.java:223)
at ome.formats.importer.cli.CommandLineImporter.start(CommandLineImporter.java:128)
at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:366)
Caused by: omero.InternalException
serverStackTrace = "ome.conditions.InternalException: Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded
"
serverExceptionClass = "ome.conditions.InternalException"
message = " Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded"
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at IceInternal.BasicStream$DynamicUserExceptionFactory.createAndThrow(BasicStream.java:2243)
at IceInternal.BasicStream.throwException(BasicStream.java:1632)
at IceInternal.Outgoing.throwUserException(Outgoing.java:442)
at omero.api._MetadataStoreDelM.populateMinMax(_MetadataStoreDelM.java:79)
at omero.api.MetadataStorePrxHelper.populateMinMax(MetadataStorePrxHelper.java:117)
at omero.api.MetadataStorePrxHelper.populateMinMax(MetadataStorePrxHelper.java:89)
at ome.formats.OMEROMetadataStoreClient.populateMinMax(OMEROMetadataStoreClient.java:2425)
... 4 more
2011-11-25 15:56:24,981 133237 [ main] INFO ome.formats.importer.ImportLibrary - Exiting on error
2011-11-25 15:56:30,945 139201 [ main] WARN ome.formats.OMEROMetadataStoreClient - Exception closing "8cfa1914-5f8a-4f77-8e63-03ad90d15804/80:2:cf:74:712078$

Re: image import python script

PostPosted: Fri Nov 25, 2011 9:40 pm
by jmoore
Hi BK,

you can either attach it to ticket:7286 or send me a private email. Information about the files would also still be appreciated.

~Josh.

Re: image import python script

PostPosted: Mon Nov 28, 2011 9:17 am
by jmoore
Hi BK,

oddly I don't see any OutOfMemory statements in the logs. Did you have to restart the server after this happened? Or did it continue functioning normally? Do these errors always happen on the same file? Or always on the same number of files?

As for the "Waiting for synchronization" lags, I'll just not that those have been removed in the 4.3 line. What does your timeline for upgrading look like? (Note: we are in the process of upgrading ASCB ourselves)

Cheers,
~Josh.

Re: image import python script

PostPosted: Mon Nov 28, 2011 3:21 pm
by bhcho
Did you have to restart the server after this happened? Or did it continue functioning normally?

I had to restart the server after this symptom.

Do these errors always happen on the same file? Or always on the same number of files?

Not always the same number of files but it seems like happening around every 1000~1500 files.

As for the "Waiting for synchronization" lags, I'll just not that those have been removed in the 4.3 line. What does your timeline for upgrading look like? (Note: we are in the process of upgrading ASCB ourselves)

Do you think this might be resolved if we upgrade our server to 4.3.x?

Best,
BK

Re: image import python script

PostPosted: Mon Nov 28, 2011 7:40 pm
by jmoore
BK,

I would assume an upgrade would fix the issue, since we've fixed many such memory leaks. However, having not seen an OutOfMemory in the logs you sent me, it's very hard to know. You might try adding the following option:
Code: Select all
-XX:-HeapDumpOnOutOfMemoryError

to the server so that we can capture what's going on.

Cheers,
~Josh.
All times are UTC
Page 2 of 2
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/