Performance issues with large ND2 files

[stelfrich]

Hi everyone,

we are experiencing performance issues with our OMERO 5.0.2 setup on Windows Server 2012. Working with large (>100GB) ND2 files even crashes OMERO.insight when trying to view a multi-channel stack.

I had a look at the other thread (http://www.openmicroscopy.org/community ... f=4&t=7400) on that topic but I am not sure if increasing the memory will solve our problem. Currently, there are no OutOfMemoryExceptions although PixelData still runs with the default settings.

I can provide more information if needed but a direction would be great so that I don't have to request all the log files.

Thanks,
Stefan

[jburel]

Hi Stefan

Sorry to hear that you have issue viewing large nd2 files. How big is a plane?
It is probably an issue in insight. Did the problem occur while viewing the image or while using the measurement tool and analysing some shapes. In the case of the measurement tool, it will certainly be a memory exception client side.
You could try to increase the memory allocated to the client to see if it fixes the issue.

Do you have access to the computer where the client was run?
If yes could you send us the insight log so we can check

Cheers

Jmarie

[stelfrich]

Hi Jean-Marie,

a single plane should be in the order of 5 to 10MB (2538px*1350px @ 16bit).

The issue (insight not being responsive) occurs when loading the viewer. However, I have just started the viewer and waited for approx. 8 minutes which results in the loading of the image. There are no measurement tools involved, just plain viewing. Once the viewer has been launched it takes about 3 to 5 minutes to switch to the next plane.

I am running the Linux client from terminal: no exceptions there (where would the log files be located, btw?).

Out of curiosity I have increased the maximum heap space for the JVM to 4GB out of which only 500MB are used when viewing the problematic ND2 file. Thus, I think the issue is on the server side.. my thoughts are, that the whole file needs to be read on the server side which could account for the 3-5 minutes waiting time during plane switching. But aren't the positions of images in ND2 files somehow cached by Bioformats for faster access?

Cheers,
Stefan

[jburel]

Hi Stefan
The loading of the plane is certainly the issue. Large planes (2kx2k) are getting more and more common but the viewers in both clients have not yet been adjusted to request section of it.
The insight client requests to render the entire plane each, this explains the time it takes to launch the viewer and also the time to retrieve the next plane or to change the rendering settings.
We have started some work to improve the support of such images with large plane. That will hopefully be available later in the year (not in the coming 5.0.3 release)

For the log file, you should have an omero folder under the user environment where the log file is stored.

Cheers

Jmarie

[stelfrich]

Thanks for the explanation, Jean-Marie!

However, I am still not really convinced that this is the problem in our case. When I open a file with the Insight client from within ImageJ, the reading of a plane doesn't seem to be the problem but what is described as analyzing of the file.. once the clients starts reading, everything seems to work fine and smooth (though way below the maximal transfer rate of the network stack).

Could this behavior really be described by the client's ability to render the plane?

Cheers,
Stefan

[rleigh]

It's not necessarily client-related since the plane is rendered server-side before being sent to the client under most circumstances, so it may be that the client is waiting on the server to load and render the plane. We have some similar size nd2 images which I will test with to try to reproduce your problem. If I can't reproduce it, it would be very useful if you would be able to provide this or another sample image which demonstrates the problem so that we can reproduce this and find what's making things so slow.


Kind regards,
Roger Leigh

[rleigh]

I have tested with a large ND2 dataset (150GB) with a large plane size, which isn't as large as yours but which should have been large enough to demonstrate the problem. I wasn't able to reproduce the slow viewing performance--the system was quite responsive. It would be great if you could provide us with a file which demonstrates the problem so we can investigate further with a known problematic dataset. Melissa will provide you with login credentials so that you may upload an example via ftp.


Many thanks,
Roger

[mlinkert]

Hi Stefan,

If you are interested in uploading an example file, I have sent FTP information in a private message.

However, I suspect that you may be running into this problem:

https://github.com/openmicroscopy/openm ... /pull/2758

which wouldn't have been taken into account during our tests. As that and several .nd2-specific performance improvements will be included in the upcoming 5.0.3 release, you may wish to try 5.0.3 before sending data to us.

Regards,
-Melissa

[stelfrich]

Sorry for the late reply..

I was digging a bit deeper. Our problem might be related to the one described in the PR (although I am not sure about the details). What I can say is that for ND2 files Memoizer files are written during import but they are never read during access. This seems to be a problem specific to ND2 files because memo files are loaded for TIFFs on the same machine..

On the other hand, we are running a small test machine also on Windows Server 2012 with default folders (on the live system we have folders in different locations) and settings. I don't observe the same problem there..

Regards,
Stefan

[sbesson]

Hi Stefan,

since the error seems to be node-specific, would it be possible for you to send us the server logs of the machine where the memo files are not properly loaded as well as a typical ND2 file showing this issue? You should be able to send both of them via http://qa.openmicroscopy.org.uk/qa/upload/ after zipping the log files.

Best,
Sebastien