We're Hiring!

Uploading large tiff genereated wrong image on OMERO 5 RC1

General user discussion about using the OMERO platform to its fullest. Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

Uploading large tiff genereated wrong image on OMERO 5 RC1

Postby bpavie » Wed Jan 29, 2014 3:27 pm

I am currently testing OMERO 5 RC1 and I was testing it with some large tiff files, and realized that there is a bug on how they are managed (looks to be that the image is stored in a duplicate pyramidal format that is wrong). The result is that I can download fine the original image but I can't visualize it properly:
viewer.png
viewer.png (69.16 KiB) Viewed 9364 times


The tiff file has been saved from a dm3 file using Fiji.
The upload of the dm3 file and the rendering using the Omero viewer is working fine.

However (and this is probably an other issue), I am not able to open both of them using the Omero plugin in Fiji. Smaller image from other dataset are however open fine using the plugin. I didn't notice any error in the Blitz-0.log file during the opening, it is just hanging for ever from the user point of view until you have to cancel it...

Let me know if you want a sample of the data.
DM3 file is an 8000x8000 16 bits file 1 channel for a size of 122 MB, same thing for the TIFF.


Benjamin
bpavie
 
Posts: 20
Joined: Fri Mar 01, 2013 5:57 pm

Re: Uploading large tiff genereated wrong image on OMERO 5 R

Postby jmoore » Thu Jan 30, 2014 10:47 am

Hi Benjamin,

A sample file uploaded to http://qa.openmicroscopy.org.uk/qa/upload/ would be ideal. If possible, could you also include the PixelData-0.log file, zipped, from var/log on the server?

Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Uploading large tiff genereated wrong image on OMERO 5 R

Postby bpavie » Thu Jan 30, 2014 12:40 pm

bpavie
 
Posts: 20
Joined: Fri Mar 01, 2013 5:57 pm

Re: Uploading large tiff genereated wrong image on OMERO 5 R

Postby jmoore » Thu Jan 30, 2014 12:48 pm

There's an OutOfMemory exception in the log file, which is fairly likely causing the corrupted pyramid file:

Code: Select all
2014-01-29 16:12:17,174 INFO  [                ome.io.nio.PixelsService] (3-thread-2) Creating BfPixelBuffer: /opt/OMERO/OMERO.data/ManagedRepository/bpavie_2/2014-01/29/16-12-03.239/Prefix_3VBSED_slice_0000.tif Series: 0
2014-01-29 16:12:17,178 INFO  [                ome.io.nio.PixelsService] (3-thread-2) Destination pyramid tile size: java.awt.Dimension[width=256,height=256]
2014-01-29 16:12:17,179 INFO  [                ome.io.nio.PixelsService] (3-thread-2) Pyramid creation for Pixels:4701 1/1024 (0%).
2014-01-29 16:12:18,564 INFO  [       loci.formats.in.MinimalTiffReader] (3-thread-2) Reading IFDs
2014-01-29 16:12:18,569 INFO  [       loci.formats.in.MinimalTiffReader] (3-thread-2) Populating metadata
2014-01-29 16:12:18,579 INFO  [              loci.formats.in.TiffReader] (3-thread-2) Checking comment style
2014-01-29 16:12:18,580 INFO  [          loci.formats.in.BaseTiffReader] (3-thread-2) Populating OME metadata
2014-01-29 16:12:19,106 ERROR [        ome.services.util.ServiceHandler] (3-thread-2) java.lang.Error:  Wrapped Exception: (java.lang.OutOfMemoryError):
Java heap space
java.lang.OutOfMemoryError: Java heap space
        at loci.formats.tiff.TiffParser.getSamples(TiffParser.java:897) ~[scifio.jar:na]
        at loci.formats.tiff.TiffParser.getSamples(TiffParser.java:745) ~[scifio.jar:na]
        at loci.formats.in.MinimalTiffReader.openBytes(MinimalTiffReader.java:292) ~[scifio.jar:na]
        at loci.formats.DelegateReader.openBytes(DelegateReader.java:219) ~[scifio.jar:na]
        at loci.formats.ImageReader.openBytes(ImageReader.java:435) ~[scifio.jar:na]
        at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:197) ~[scifio.jar:na]
        at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:226) ~[scifio.jar:na]
        at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:332) ~[scifio.jar:na]
        at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:332) ~[scifio.jar:na]
        at loci.formats.MinMaxCalculator.openBytes(MinMaxCalculator.java:273) ~[scifio.jar:na]
        at ome.io.bioformats.BfPixelsWrapper.getTile(BfPixelsWrapper.java:345) ~[romio.jar:na]


The PixelData service is still at the default 256 MB:
Code: Select all
2014-01-29 16:02:55,932 INFO  [      ome.services.util.JvmSettingsCheck] (      main) Max Memory (MB):   =    227
2014-01-29 16:02:55,936 INFO  [      ome.services.util.JvmSettingsCheck] (      main) OS Memory (MB):    =   7983


Could you spare up to 1GB for it? See http://www.openmicroscopy.org/site/support/omero4/sysadmins/unix/server-installation.html#jvm-memory-settings for instructions.

Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Uploading large tiff genereated wrong image on OMERO 5 R

Postby bpavie » Fri Jan 31, 2014 2:24 pm

Thanks, that solve the problem.
I did the following modification in the OMERO.server/etc/grid/templates.xml

Code: Select all
<server-template id="PixelDataTemplate">
      <parameter name="index"/>
      <parameter name="dir"/>
      <parameter name="config" default="default"/>
      <server id="PixelData-${index}" exe="${JAVA}" activation="always" pwd="${OMERO_HOME}">
        <target name="heap-dump">
            <option>-XX:+HeapDumpOnOutOfMemoryError</option>
        </target>
        <target name="heap-dump-tmp">
            <option>-XX:+HeapDumpOnOutOfMemoryError</option>
            <option>-XX:HeapDumpPath=/tmp</option>
        </target>
        <!-- <option>-Xmx256M</option> -->
        <option>-Xmx1024M</option>

It would be nice to describe the following more in details in the documentation. I didn't fully realized that they was multiple service (Blitz, PixelData, Indexer and Repository), each one with their own memory setting, so I modified only the Blitz one... Is that also recommended to increase the memory for the indexer and the repository ?

Thanks,
Benjamin
bpavie
 
Posts: 20
Joined: Fri Mar 01, 2013 5:57 pm

Re: Uploading large tiff genereated wrong image on OMERO 5 R

Postby Sethur » Sat Feb 01, 2014 12:59 pm

Hi Benjamin & Josh,

I agree on this one, this has to be included in the documentation in more detail. Showing a reg-ex search and replace command is not enough for sys admins to even realize that there is more than one memory setting. Also, there has to be a description of what makes sense in small, medium or large user environments, etc. and what impact the settings have on the functioning of the OMERO server.

Personally, I did not use the rex-ex replace in the beginning and only now realized that I missed the memory settings for all the other services besides Blitz.

Cheers,

Tristan
Sethur
 
Posts: 112
Joined: Thu Jan 16, 2014 11:34 pm

Re: Uploading large tiff genereated wrong image on OMERO 5 R

Postby sbesson » Tue Feb 04, 2014 9:22 am

Hi Tristan and Benjamin,

thanks for all the valuable feedback on this thread. We are agree our sysadmin documentation still lacks many details like the description of the server templates. An initial documentation Pull Request is currently under examination to improve its Advanced configuration section.

Regarding other server instances (Repository, Indexer), according to your logs, there is no reason to increase their default memory settings.

Best regards,
Sebastien
User avatar
sbesson
Team Member
 
Posts: 421
Joined: Tue Feb 28, 2012 7:20 pm


Return to User Discussion

Who is online

Users browsing this forum: No registered users and 0 guests