We're Hiring!

Local OMERO server down?

General user discussion about using the OMERO platform to its fullest. Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

Local OMERO server down?

Postby ferranda » Fri May 18, 2012 1:06 pm

Hi,

We implemented our facility with OMERO a few months ago, using the OMERO virtual machine. We have a few test users using it now.

Everything is going smoothly, and we have good feedbach from users.

Unfortunately, today we are experiencing an issue on our server. We cannot retrieve data, or import new ones. But we can see the tree of data for each user. The user was working with a large amount of data yesterday and only today we cannot retrieve the data.

The cache might have run out of memory? Or we lost the connection somehow?
We have different error messages depending on the action we do: retrieving data or importing new data. See below for the error messages.

We are using OMERO 4.3.3.

Any tips to try to fix this problem, please?

Thanks!



The error log when trying to retrieve data:

java.lang.Exception: org.openmicroscopy.shoola.env.data.DSAccessException: Cannot access data.
Cannot load hierarchy for class pojos.DatasetData.
at org.openmicroscopy.shoola.env.data.OMEROGateway.handleException(OMEROGateway.java:911)
at org.openmicroscopy.shoola.env.data.OMEROGateway.loadContainerHierarchy(OMEROGateway.java:2655)
at org.openmicroscopy.shoola.env.data.OmeroDataServiceImpl.loadContainerHierarchy(OmeroDataServiceImpl.java:203)
at org.openmicroscopy.shoola.env.data.views.calls.DMLoader$1.doCall(DMLoader.java:87)
at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:274)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Thread.java:680)
Caused by: omero.InternalException
serverStackTrace = "ome.conditions.InternalException: Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space
"
serverExceptionClass = "ome.conditions.InternalException"
message = " Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space"
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at IceInternal.BasicStream$DynamicUserExceptionFactory.createAndThrow(BasicStream.java:2243)
at IceInternal.BasicStream.throwException(BasicStream.java:1632)
at IceInternal.Outgoing.throwUserException(Outgoing.java:442)
at omero.api._IContainerDelM.loadContainerHierarchy(_IContainerDelM.java:592)
at omero.api.IContainerPrxHelper.loadContainerHierarchy(IContainerPrxHelper.java:730)
at omero.api.IContainerPrxHelper.loadContainerHierarchy(IContainerPrxHelper.java:702)
at org.openmicroscopy.shoola.env.data.OMEROGateway.loadContainerHierarchy(OMEROGateway.java:2651)
... 9 more

at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.showErrorDialog(UserNotifierImpl.java:182)
at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.notifyError(UserNotifierImpl.java:271)
at org.openmicroscopy.shoola.env.ui.UserNotifierImpl.notifyError(UserNotifierImpl.java:241)
at org.openmicroscopy.shoola.agents.treeviewer.DataBrowserLoader.handleException(DataBrowserLoader.java:141)
at org.openmicroscopy.shoola.env.data.events.DSCallAdapter.eventFired(DSCallAdapter.java:84)
at org.openmicroscopy.shoola.env.data.views.BatchCallMonitor$1.run(BatchCallMonitor.java:124)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:682)
at java.awt.EventQueue.access$000(EventQueue.java:85)
at java.awt.EventQueue$1.run(EventQueue.java:643)
at java.awt.EventQueue$1.run(EventQueue.java:641)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:652)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:296)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:211)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:196)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:188)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)


The error message when trying to import new data:

Ice.UnknownLocalException
unknown = "TcpTransceiver.cpp:100: Ice::ConnectionLostException:
connection lost: Connection reset by peer"
at IceInternal.Outgoing.invoke(Outgoing.java:147)
at omero.api._RawPixelsStoreDelM.setTile(_RawPixelsStoreDelM.java:1767)
at omero.api.RawPixelsStorePrxHelper.setTile(RawPixelsStorePrxHelper.java:2435)
at omero.api.RawPixelsStorePrxHelper.setTile(RawPixelsStorePrxHelper.java:2407)
at ome.formats.OMEROMetadataStoreClient.setTile(OMEROMetadataStoreClient.java:2522)
at ome.formats.importer.ImportLibrary.writeDataTileBased(ImportLibrary.java:884)
at ome.formats.importer.ImportLibrary.importData(ImportLibrary.java:796)
at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:573)
at org.openmicroscopy.shoola.env.data.OMEROGateway.importImage(OMEROGateway.java:6402)
at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1228)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:85)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:54)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:111)
at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:274)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Unknown Source)

at org.openmicroscopy.shoola.env.data.OMEROGateway.importImage(OMEROGateway.java:6447)
at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1228)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:85)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:54)
at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:111)
at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:274)
at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Unknown Source) Caused by: Ice.UnknownLocalException
unknown = "TcpTransceiver.cpp:100: Ice::ConnectionLostException:
connection lost: Connection reset by peer"
at IceInternal.Outgoing.invoke(Outgoing.java:147)
at omero.api._RawPixelsStoreDelM.setTile(_RawPixelsStoreDelM.java:1767)
at omero.api.RawPixelsStorePrxHelper.setTile(RawPixelsStorePrxHelper.java:2435)
at omero.api.RawPixelsStorePrxHelper.setTile(RawPixelsStorePrxHelper.java:2407)
at ome.formats.OMEROMetadataStoreClient.setTile(OMEROMetadataStoreClient.java:2522)
at ome.formats.importer.ImportLibrary.writeDataTileBased(ImportLibrary.java:884)
at ome.formats.importer.ImportLibrary.importData(ImportLibrary.java:796)
at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:573)
at org.openmicroscopy.shoola.env.data.OMEROGateway.importImage(OMEROGateway.java:6402)
... 11 more
ferranda
 
Posts: 3
Joined: Fri May 18, 2012 11:39 am

Re: Local OMERO server down?

Postby jmoore » Fri May 18, 2012 1:42 pm

How much memory have you given to your virtual machine? Have you modified the settings for OMERO at all? Do you remember what action you were performing when it ran out of memory?

Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Local OMERO server down?

Postby ferranda » Sun May 20, 2012 6:10 am

Hi Josh,

I am not sure how much memory was assigned in the end. When I first installed the virtual machine, I did not modify the settings at all. Then we had to reinstall it later on, and my colleague is now in charge of OMERO, and he probably changed some settings, as we wanted to have it to a bigger scale.
Sorry I cannot be more precise...

The user was adjusting the intensity levels of about 200 dv files and using the batch processing. The progress window appeared then he had the error message that the data is not accessible (see first post).
From there, he could see his data tree (folders), but no thumbnails or images in the folders.

My colleagues in charge of this side of OMERO is back on Monday, I'll check with him.

Cheers,
Alexia
ferranda
 
Posts: 3
Joined: Fri May 18, 2012 11:39 am

Re: Local OMERO server down?

Postby ehrenfeu » Mon May 21, 2012 1:58 pm

Hey Josh,

the machine still runs as it was preconfigured in the VirtualBox image, which means it has 1 GB of RAM (we just converted it into a VMware/ESX image to deploy it on our infrastructure here).

Modifications in the OMERO settings were just made for the storage location and the LDAP/ActiveDirectory authentication, nothing else was changed. More precise, the java processes still run with the "-Xmx512M" flag if this is your question ;)

I don't know for sure what the user did exactly when this error showed up, but I will try to reproduce it with him - yet this will take some time as he is rather busy preparing a talk... As Alexia has already mentioned, he probably tried to batch process some of his files, but we'll see.

Anything useful I can figure out? If it helps, I can attach the logfiles (or use gist if you prefer).
Cheers,
Niko
User avatar
ehrenfeu
 
Posts: 90
Joined: Fri May 11, 2012 8:21 am
Location: Basel, Switzerland

Re: Local OMERO server down?

Postby cxallan » Tue May 22, 2012 12:30 pm

At a minimum it would be good to check the disk space allocated to ensure that it hasn't filled up. Failing an issue with that the output of bin/omero admin diagnostics would be a useful first start.
cxallan
Site Admin
 
Posts: 509
Joined: Fri May 01, 2009 8:07 am

Re: Local OMERO server down?

Postby ehrenfeu » Tue May 22, 2012 12:35 pm

The disk space definitely hasn't filled up. The repository storage has ~250GB free space, the root file system more than 25GB.

Here's the diag output:
Code: Select all
================================================================================
OMERO Diagnostics 4.3.3-1155-ac713a4-ice33-b2964
================================================================================

WARNING:omero.util.UpgradeCheck:UPGRADE AVAILABLE:Please upgrade to Beta-4.3.4 See http://trac.openmicroscopy.org.uk/omero for the latest version

Commands:   java -version                  1.6.0     (/usr/bin/java)
Commands:   python -V                      2.6.6     (/usr/bin/python)
Commands:   icegridnode --version          3.3.1     (/usr/bin/icegridnode)
Commands:   icegridadmin --version         3.3.1     (/usr/bin/icegridadmin)
Commands:   psql --version                 8.4.8     (/usr/bin/psql -- 2 others)

Server:     icegridnode                    running
Server:     Blitz-0                        active (pid = 20799, enabled)
Server:     DropBox                        active (pid = 20818, enabled)
Server:     FileServer                     active (pid = 20819, enabled)
Server:     Indexer-0                      inactive (disabled)
Server:     MonitorServer                  active (pid = 20821, enabled)
Server:     OMERO.Glacier2                 active (pid = 20822, enabled)
Server:     OMERO.IceStorm                 active (pid = 20823, enabled)
Server:     PixelData-0                    inactive (disabled)
Server:     Processor-0                    active (pid = 20825, enabled)
Server:     Tables-0                       active (pid = 20826, enabled)
Server:     TestDropBox                    inactive (enabled)

Log dir:    /home/omero/OMERO.server/var/log exists

Log files:  Blitz-0.log                    16.0 MB       errors=4    warnings=62
Log files:  Blitz-0.log.1                  512.0 MB      errors=479  warnings=711
Log files:  DropBox.log                    14.0 KB       errors=0    warnings=5
Log files:  FileServer.log                 2.0 KB
Log files:  Indexer-0.log                  208.0 KB      errors=7    warnings=0
Log files:  MonitorServer.log              9.0 KB        errors=0    warnings=4
Log files:  OMEROweb.log                   0.0 KB
Log files:  OMEROweb_request.log           0.0 KB
Log files:  PixelData-0.log                211.0 KB      errors=7    warnings=0
Log files:  Processor-0.log                2.0 MB        errors=1    warnings=1
Log files:  Tables-0.log                   9.0 KB        errors=0    warnings=2
Log files:  TestDropBox.log                n/a
Log files:  master.err                     351.0 KB
Log files:  master.out                     0.0 KB
Log files:  Total size                     531.97 MB


Environment:OMERO_HOME=(unset)
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/lib/jvm/java-6-sun/bin:/usr/lib/jvm/java-6-sun/bin:/usr/share/Ice-3.3.1:/usr/lib/postgresql/8.4/bin:/home/omero/OMERO.server/bin
Environment:ICE_HOME=/usr/share/Ice-3.3.1
Environment:LD_LIBRARY_PATH=/usr/share/java:/usr/lib:
Environment:DYLD_LIBRARY_PATH=/usr/share/java:/usr/lib:

OMERO data dir: '/export/OMERO.data/'   Exists? True    Is writable? True
OMERO.web status... [RUNNING] (PID 6643)
User avatar
ehrenfeu
 
Posts: 90
Joined: Fri May 11, 2012 8:21 am
Location: Basel, Switzerland

Re: Local OMERO server down?

Postby cxallan » Tue May 22, 2012 12:42 pm

All right that's one issue down then. Can you ZIP up your var/log/Blitz-0.log and attach it here please?
cxallan
Site Admin
 
Posts: 509
Joined: Fri May 01, 2009 8:07 am

Re: Local OMERO server down?

Postby ehrenfeu » Tue May 22, 2012 12:49 pm

I'll strip it down to the relevant date and try to attach it here. What filename suffixes are allowed for attachments btw? (log and txt are not...)
User avatar
ehrenfeu
 
Posts: 90
Joined: Fri May 11, 2012 8:21 am
Location: Basel, Switzerland

Re: Local OMERO server down?

Postby cxallan » Tue May 22, 2012 1:14 pm

Just ZIP it up, those are fine.
cxallan
Site Admin
 
Posts: 509
Joined: Fri May 01, 2009 8:07 am

Re: Local OMERO server down?

Postby ehrenfeu » Tue May 29, 2012 10:08 am

Sorry, but I've been pretty busy the last days...

In addition, the board software allows just for file sizes of max. 256 KB, so eventually I placed the 7-zip compressed logfile in the download space of our xuvtools project:

http://data.xuvtools.org/tmp/Blitz-0_2012-05-17.log.7z

You can find the corresponding error message in line 495460 (or using the timestamp "2012-05-17 18:46:49,369")

Cheers
~Niko
User avatar
ehrenfeu
 
Posts: 90
Joined: Fri May 11, 2012 8:21 am
Location: Basel, Switzerland

Next

Return to User Discussion

Who is online

Users browsing this forum: No registered users and 0 guests