Omero crashes under use with GC overhead limit exceeded

[Magic]

We finally made the decision to start with a brand new database on Omero. Everything was purged completely and recreated.

Things seemed to work okay for a very short while. But now we have an issue that is causing the system to crash. It seems to be triggered by viewing an image, but not necessarily the same image.

Here's the error message from the logs:

**************

2013-09-21 02:37:27,938 WARNI [ omero.gateway] (proc.09827) debug:3439 InternalException on <class 'omeroweb.webclient.webclient_gateway.OmeroWebSafeCallWrapper'> to <73fc6216-de28-4f68-9aa6-6bef40186a56omero.api.RenderingEngine> renderCompressed((object #0 (::omero::romio::PlaneDef)
{
slice = 0
x = 0
y = 0
z = 21
t = 0
region = <nil>
stride = 0
}, <ServiceOptsDict: {'omero.session.uuid': '19635879-2080-4480-801c-4f3da2086df3', 'omero.group': '-1', 'omero.client.uuid': '6b85f9d2-d893-4064-98c6-46b93d5c2a13'}>), {})
Traceback (most recent call last):
File "/home/vali/apps/OMERO/OMERO.server/lib/python/omero/gateway/__init__.py", line 3457, in __call__
return self.f(*args, **kwargs)
File "/home/vali/apps/OMERO/OMERO.server/lib/python/omero_api_RenderingEngine_ice.py", line 326, in renderCompressed
return _M_omero.api.RenderingEngine._op_renderCompressed.invoke(self, ((_def, ), _ctx))
InternalException: exception ::omero::InternalException
{
serverStackTrace = ome.conditions.InternalException: Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded

serverExceptionClass = ome.conditions.InternalException
message = Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded
}
2013-09-21 02:37:27,940 WARNI [ omeroweb.feedback.views] (proc.09827) handler404:162 Not Found: /omero/webclient/render_image/34/21/0/
2013-09-21 02:37:28,718 INFO [ omero.gateway] (proc.09828) connect:1708 created connection (uuid=19635879-2080-4480-801c-4f3da2086df3)

***********

What do you recommend? Is any more info needed? The system has plenty of drive space and 32gb of RAM and does nothing but omero, so it's not as if it could have run out of memory. (Ubuntu 12.04)

Thanks!

[jmoore]

Have you increased the memory available to OMERO? If not, take a look at the JVM memory settings section.

Cheers,
~Josh

[Magic]

Thanks for your quick response.

I increased the memory limits from -Xmx512M to -Xmx4096M

And also from -XX:MaxPermSize=128m to -XX:MaxPermSize=512M

(Notice the switch from a lowercase m to a capital M following the documentation. Is that correct? I assume it doesn't matter if it's lowercase or capital).

I then restarted omero.

Is this not enough memory? If it should be something else can you tell me exactly what it should be for a 32GB RAM system.

Things broke today when uploading a tiff file of approximately 1GB.

This is the error message the client showed: http://penguinpython.com/omero/omero_crash.png

After that it was impossible to login via the client: http://penguinpython.com/omero/omero_login_fail.png

I didn't notice anything immediately strange in the logs, but since the screenshot shows the exact import time and date maybe I should tar them up and send them via email. If so, let me know exactly what logs or directories of logs you'd find helpful.

Thanks!

[Magic]

Josh...

Given the time zone difference I'm going to be pro-active and email the logs to you as a tar file. I hope that's okay.

I'll include all of the omero logs in ~/apps/OMERO/OMERO.server/var/log and also the postgresql logs.

I'll also attach the output of omero admin diagnostics.

Let me know if there is anything else that would be useful to include.

Thanks
Sam

[mtbc]

Sam, just to let you know that Josh is tied up in meetings in London today, so he might take a day to get back to you on the e-mailed logs. With luck, they'll make it obvious how your problem can be fixed.

Cheers,
Mark

[Magic]

Thanks Mark and Josh. I appreciate the response, and we'll be waiting with our fingers crossed.

[jmoore]

Hi Sam,

in your logs, the last GC related error were on the 20th in Blitz-0.log:

Code: Select all

2013-09-20 18:37:52,997 INFO  [        ome.services.util.ServiceHandler] (l.Server-9)  Excp:    ome.conditions.InternalException:  Wrapped Exception:
(java.lang.OutOfMemoryError):
GC overhead limit exceeded


and then in OMEROweb.log on the 21st:

Code: Select all

2013-09-21 02:37:27,938 WARNI [                           omero.gateway] (proc.09827) debug:3439 InternalException ... Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded


which are both before your last server restart:

Code: Select all

$ grep Ready Blitz-0.log
2013-09-18 16:42:00,994 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 4.4.8p1-ice34-b304 Ready.
2013-09-20 18:03:25,368 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 4.4.8p1-ice34-b304 Ready.
2013-09-20 18:31:22,452 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 4.4.8p1-ice34-b304 Ready.
2013-09-20 18:56:49,147 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 4.4.8p1-ice34-b304 Ready.
2013-09-21 11:46:51,142 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 4.4.8p1-ice34-b304 Ready.
2013-09-23 13:04:12,955 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 4.4.8p1-ice34-b304 Ready.


In other words, I would think you're latest issues are something different. Would you mind restarting your server (bin/omero admin restart), attempting to login in via insight, and send us the insight log on failure?

Thanks,
~Josh.

[Magic]

Things seem to be working today so far, so that's the good news.

I realized some of the clients we had on various computers were out of date. Maybe that was part of the problem.

Have you thought about adding a notice upon start-up of the client telling the user a new version is available?

Thanks for the help.

[Magic]

Josh...

Unfortunately we've seen more breakage. I'll forward you via email the following:

1) A description of the problem a user has and a screenshot
2) The client logfile from the person above
3) Another client logfile from someone having trouble at around the same time today
4) The latest server logfiles

Let me know if there's anything else you need to diagnose the problem.

Thanks

[jmoore]

Hi Sam,

thanks for the new logs. It seems you're still having memory issues on the server:

Code: Select all

2013-09-26 14:31:40,522 ERROR [        ome.services.util.ServiceHandler] (.Server-17) java.lang.Error:  Wrapped Exception: (java.lang.OutOfMemoryError):
2013-09-26 14:31:40,523 WARN  [        ome.services.util.ServiceHandler] (.Server-17) Unknown exception thrown.
2013-09-26 14:31:42,514 ERROR [        ome.services.util.ServiceHandler] (.Server-25) java.lang.Error:  Wrapped Exception: (java.lang.OutOfMemoryError):
...


Could you paste the output of "jps -v" here?

Thanks,
~Josh