We're Hiring!

Omero crashes under use with GC overhead limit exceeded

Having a problem deploying OMERO? Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

The OMERO.server installation documentation begins here and you can find OMERO.web deployment documentation here.

Omero crashes under use with GC overhead limit exceeded

Postby Magic » Sat Sep 21, 2013 1:53 am

We finally made the decision to start with a brand new database on Omero. Everything was purged completely and recreated.

Things seemed to work okay for a very short while. But now we have an issue that is causing the system to crash. It seems to be triggered by viewing an image, but not necessarily the same image.

Here's the error message from the logs:

**************

2013-09-21 02:37:27,938 WARNI [ omero.gateway] (proc.09827) debug:3439 InternalException on <class 'omeroweb.webclient.webclient_gateway.OmeroWebSafeCallWrapper'> to <73fc6216-de28-4f68-9aa6-6bef40186a56omero.api.RenderingEngine> renderCompressed((object #0 (::omero::romio::PlaneDef)
{
slice = 0
x = 0
y = 0
z = 21
t = 0
region = <nil>
stride = 0
}, <ServiceOptsDict: {'omero.session.uuid': '19635879-2080-4480-801c-4f3da2086df3', 'omero.group': '-1', 'omero.client.uuid': '6b85f9d2-d893-4064-98c6-46b93d5c2a13'}>), {})
Traceback (most recent call last):
File "/home/vali/apps/OMERO/OMERO.server/lib/python/omero/gateway/__init__.py", line 3457, in __call__
return self.f(*args, **kwargs)
File "/home/vali/apps/OMERO/OMERO.server/lib/python/omero_api_RenderingEngine_ice.py", line 326, in renderCompressed
return _M_omero.api.RenderingEngine._op_renderCompressed.invoke(self, ((_def, ), _ctx))
InternalException: exception ::omero::InternalException
{
serverStackTrace = ome.conditions.InternalException: Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded

serverExceptionClass = ome.conditions.InternalException
message = Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded
}
2013-09-21 02:37:27,940 WARNI [ omeroweb.feedback.views] (proc.09827) handler404:162 Not Found: /omero/webclient/render_image/34/21/0/
2013-09-21 02:37:28,718 INFO [ omero.gateway] (proc.09828) connect:1708 created connection (uuid=19635879-2080-4480-801c-4f3da2086df3)

***********

What do you recommend? Is any more info needed? The system has plenty of drive space and 32gb of RAM and does nothing but omero, so it's not as if it could have run out of memory. (Ubuntu 12.04)

Thanks!
Magic
 
Posts: 19
Joined: Thu Sep 12, 2013 10:56 pm

Re: Omero crashes under use with GC overhead limit exceeded

Postby jmoore » Sat Sep 21, 2013 12:24 pm

Have you increased the memory available to OMERO? If not, take a look at the JVM memory settings section.

Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Omero crashes under use with GC overhead limit exceeded

Postby Magic » Tue Sep 24, 2013 12:33 am

Thanks for your quick response.

I increased the memory limits from -Xmx512M to -Xmx4096M

And also from -XX:MaxPermSize=128m to -XX:MaxPermSize=512M

(Notice the switch from a lowercase m to a capital M following the documentation. Is that correct? I assume it doesn't matter if it's lowercase or capital).

I then restarted omero.

Is this not enough memory? If it should be something else can you tell me exactly what it should be for a 32GB RAM system.

Things broke today when uploading a tiff file of approximately 1GB.

This is the error message the client showed: http://penguinpython.com/omero/omero_crash.png

After that it was impossible to login via the client: http://penguinpython.com/omero/omero_login_fail.png

I didn't notice anything immediately strange in the logs, but since the screenshot shows the exact import time and date maybe I should tar them up and send them via email. If so, let me know exactly what logs or directories of logs you'd find helpful.

Thanks!
Magic
 
Posts: 19
Joined: Thu Sep 12, 2013 10:56 pm

Re: Omero crashes under use with GC overhead limit exceeded

Postby Magic » Tue Sep 24, 2013 1:01 am

Josh...

Given the time zone difference I'm going to be pro-active and email the logs to you as a tar file. I hope that's okay.

I'll include all of the omero logs in ~/apps/OMERO/OMERO.server/var/log and also the postgresql logs.

I'll also attach the output of omero admin diagnostics.

Let me know if there is anything else that would be useful to include.

Thanks
Sam
Magic
 
Posts: 19
Joined: Thu Sep 12, 2013 10:56 pm

Re: Omero crashes under use with GC overhead limit exceeded

Postby mtbc » Tue Sep 24, 2013 9:02 am

Sam, just to let you know that Josh is tied up in meetings in London today, so he might take a day to get back to you on the e-mailed logs. With luck, they'll make it obvious how your problem can be fixed.

Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Re: Omero crashes under use with GC overhead limit exceeded

Postby Magic » Tue Sep 24, 2013 6:28 pm

Thanks Mark and Josh. I appreciate the response, and we'll be waiting with our fingers crossed. :)
Magic
 
Posts: 19
Joined: Thu Sep 12, 2013 10:56 pm

Re: Omero crashes under use with GC overhead limit exceeded

Postby jmoore » Wed Sep 25, 2013 8:52 am

Hi Sam,

in your logs, the last GC related error were on the 20th in Blitz-0.log:
Code: Select all
2013-09-20 18:37:52,997 INFO  [        ome.services.util.ServiceHandler] (l.Server-9)  Excp:    ome.conditions.InternalException:  Wrapped Exception:
(java.lang.OutOfMemoryError):
GC overhead limit exceeded

and then in OMEROweb.log on the 21st:
Code: Select all
2013-09-21 02:37:27,938 WARNI [                           omero.gateway] (proc.09827) debug:3439 InternalException ... Wrapped Exception: (java.lang.OutOfMemoryError):
GC overhead limit exceeded


which are both before your last server restart:
Code: Select all
$ grep Ready Blitz-0.log
2013-09-18 16:42:00,994 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 4.4.8p1-ice34-b304 Ready.
2013-09-20 18:03:25,368 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 4.4.8p1-ice34-b304 Ready.
2013-09-20 18:31:22,452 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 4.4.8p1-ice34-b304 Ready.
2013-09-20 18:56:49,147 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 4.4.8p1-ice34-b304 Ready.
2013-09-21 11:46:51,142 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 4.4.8p1-ice34-b304 Ready.
2013-09-23 13:04:12,955 INFO  [    ome.services.util.ServerVersionCheck] (      main) OMERO Version: 4.4.8p1-ice34-b304 Ready.


In other words, I would think you're latest issues are something different. Would you mind restarting your server (bin/omero admin restart), attempting to login in via insight, and send us the insight log on failure?

Thanks,
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Omero crashes under use with GC overhead limit exceeded

Postby Magic » Thu Sep 26, 2013 12:29 am

Things seem to be working today so far, so that's the good news.

I realized some of the clients we had on various computers were out of date. Maybe that was part of the problem.

Have you thought about adding a notice upon start-up of the client telling the user a new version is available?

Thanks for the help.
Magic
 
Posts: 19
Joined: Thu Sep 12, 2013 10:56 pm

Re: Omero crashes under use with GC overhead limit exceeded

Postby Magic » Thu Sep 26, 2013 11:27 pm

Josh...

Unfortunately we've seen more breakage. I'll forward you via email the following:

1) A description of the problem a user has and a screenshot
2) The client logfile from the person above
3) Another client logfile from someone having trouble at around the same time today
4) The latest server logfiles

Let me know if there's anything else you need to diagnose the problem.

Thanks
Magic
 
Posts: 19
Joined: Thu Sep 12, 2013 10:56 pm

Re: Omero crashes under use with GC overhead limit exceeded

Postby jmoore » Fri Sep 27, 2013 6:52 am

Hi Sam,

thanks for the new logs. It seems you're still having memory issues on the server:
Code: Select all
2013-09-26 14:31:40,522 ERROR [        ome.services.util.ServiceHandler] (.Server-17) java.lang.Error:  Wrapped Exception: (java.lang.OutOfMemoryError):
2013-09-26 14:31:40,523 WARN  [        ome.services.util.ServiceHandler] (.Server-17) Unknown exception thrown.
2013-09-26 14:31:42,514 ERROR [        ome.services.util.ServiceHandler] (.Server-25) java.lang.Error:  Wrapped Exception: (java.lang.OutOfMemoryError):
...


Could you paste the output of "jps -v" here?

Thanks,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Next

Return to Installation and Deployment

Who is online

Users browsing this forum: No registered users and 1 guest