Omero crashes under use with GC overhead limit exceeded

[Magic]

Josh...

If you can tell me exactly what the optimal memory settings should be for a system with 32 GB of RAM that would be awesome.

Here's the output of jps -v

24792 blitz.jar -Xmx256M -Djava.awt.headless=true -Dlog4j.configuration=etc/log4j-indexing.xml -Domero.logfile=var/log/${omero.name}.log -Domero.name=Indexer-0
30217 Jps -Dapplication.home=/usr/lib/jvm/java-7-openjdk-amd64 -Xms8m
24760 blitz.jar -Xmx4096M -XX:MaxPermSize=512M -Djava.awt.headless=true -Dlog4j.configuration=etc/log4j.xml -Domero.logfile=var/log/${omero.name}.log -Domero.name=Blitz-0
24816 blitz.jar -Xmx256M -Djava.awt.headless=true -Dlog4j.configuration=etc/log4j-indexing.xml -Domero.logfile=var/log/${omero.name}.log -Domero.name=PixelData-0

Thank you!
-Sam

[jmoore]

Hi Sam,

4GB should certainly be enough for most uses, and that's reflected in your jps output. Do you have a rough idea of what was being uploaded? Number of images, their sizes, etc?

~Josh.

[jburel]

Hi

We do check if there is an update, but we do not bring up the dialog.
We checked if the client version and server version match, depending on the results we should indicate to either update the client or both.

Jmarie

[Magic]

Josh...

Most of the problems recently seem to be when two or three people are using it. We may have had as many as five people using the system at a time. Originally there were lots of problems even with just one user, but I'm not sure that's still the case (although it might be).

I get the feeling people are uploading lots of files that they've accumulated while waiting for Omero to be ready for them.

I don't really know how large or what type of files have been uploaded. But just looking at the Omero.data/Files directory I see the largest file is 10MB and most of the files are 2.5MB or less. I can get more information tomorrow when everyone is back in the lab, but I'd love to try and fix it tonight for them if you have any other suggestions.

I did ask whether it seemed to fail on the same file every time, and was told it didn't. (Obviously if we had a file that always caused trouble, it would make the debugging much easier).

Thanks
Sam

[jmoore]

Sam,

Those user counts and those file sizes should really not be causing any issues with 4GB of RAM. You could of course just try more, say 8GB or even 16GB. But that would certainly point to something quite unusual in the server it'd be great if you could help us pinpoint, whether a leak or a file format that's opening a large byte array.

Sorry I couldn't be of more help. Has anyone else on the lists seen similar behavior?

Cheers,
~Josh

[Magic]

Okay I've asked the team members for more information about file sizes/formats. I'll post them as I get responses. Here's the first response:

"This may not directly answer your question but in my usage, it hasn't mattered what the file size is. The crashing seems correlated to instances where multiple people are logged on and using Omero simultaneously. I have had very few issues when I am the only person on Omero."

I'm curious, how many clients have you tested simultaneous uploading with?

Thanks

[Magic]

A second user report: "I don't notice a pattern of file types causing stalling of the server. Generically, I upload a folder of size 100MB to 700MB. The Folder has files of 100MB to 300MB each, and sometimes it has single images of 5-10MB. The files that are 100MB to 300MB are stacks (series) of many images, and each image is again 5-10MB."

Any thoughts as to what to do next?

[jmoore]

Hi Sam,

Would it be possible to reproduce with

Code: Select all

-XX:+HeapDumpOnOutOfMemoryError

enabled on the server? This will likely produce a very large dump file that it would be good to have access to, either via download or jhat.

Thanks for helping us to track this down.
~Josh.

[Magic]

Josh...

Sounds like a great idea. Can you tell me exactly where to add this and I'll put it in tomorrow.

Thanks!

[jmoore]

Along side your -Xmx setting in templates.xml add:

Code: Select all

<option>-XX:+HeapDumpOnOutOfMemoryError</option>


The generated hprof file should be at the top-level of your OMERO installation:

Code: Select all

$ ls -ltra
total 18500
drwxrwxr-x  3 jamoore jamoore     4096 Oct  1 08:22 include
drwxrwxr-x  2 jamoore jamoore     4096 Oct  1 08:22 bin
-rw-rw-r--  1 jamoore jamoore    17987 Oct  1 08:22 LICENSE.txt
drwxrwxr-x  2 jamoore jamoore     4096 Oct  1 08:22 licenses
drwxrwxr-x  4 jamoore jamoore     4096 Oct  1 08:22 etc
drwxrwxr-x  4 jamoore jamoore     4096 Oct  1 08:22 sql
drwxrwxr-x  7 jamoore jamoore     4096 Oct  1 08:22 lib
drwxrwxr-x 15 jamoore jamoore     4096 Oct  2 10:09 ..
drwxrwxr-x  5 jamoore jamoore     4096 Oct  2 10:12 var
drwxrwxr-x  9 jamoore jamoore     4096 Oct  2 10:13 .
-rw-------  1 jamoore jamoore 18885374 Oct  2 10:13 java_pid7995.hprof


If you don't think there will be enough space for the ~4GB dump, you can point it somewhere else with:

Code: Select all

<option>-XX:HeapDumpPath=/tmp</option>


For example:

Code: Select all

$ ls -ltra /tmp
...
-rw-------  1 jamoore jamoore 22196368 Oct  2 10:19 java_pid9013.hprof


I've opened a PR with these settings available so you can see them in action: https://github.com/openmicroscopy/openmicroscopy/pull/1570

Cheers,
~Josh