We're Hiring!

pyramid creation on large image - out of memory

General user discussion about using the OMERO platform to its fullest. Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

pyramid creation on large image - out of memory

Postby dsudar » Mon Oct 16, 2017 8:34 pm

Hi,

I'm trying to import very large images (over 100K by 100K pixels) generated by FEI Maps software in TIFF format. They initially import fine, the pyramid creation appears to complete fine, but then I get a set of out of memory errors in PixelData-0.log:
Code: Select all
2017-10-16 08:36:28,034 INFO  [             ome.io.nio.FilePathResolver] (2-thread-3) Metadata only file, resulting path: /data/share/OMERO.data/ManagedRepository/riesterj_1653/2017-10/16/08-30-12.652/SMMART 101b-1.tif
2017-10-16 08:36:28,043 INFO  [             ome.io.nio.FilePathResolver] (2-thread-5) Metadata only file, resulting path: /data/share/OMERO.data/ManagedRepository/riesterj_1653/2017-10/16/08-30-12.652/SMMART 101b-1.tif
2017-10-16 08:36:28,043 WARN  [ ome.services.pixeldata.PixelDataHandler] (2-thread-5) Pixels:812259 -- Already locked! /data/share/OMERO.data/Pixels/Dir-812/.812259_pyramid.pyr_lock
2017-10-16 08:36:28,064 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Creating BfPixelBuffer: /data/share/OMERO.data/ManagedRepository/riesterj_1653/2017-10/16/08-30-12.652/SMMART 101b-1.tif Series: 0
2017-10-16 08:36:28,064 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Destination pyramid tile size: java.awt.Dimension[width=512,height=512]
2017-10-16 08:36:28,064 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 1/140400 (0%).
2017-10-16 08:51:22,555 INFO  [                      ome.system.metrics] (r-thread-1) type=TIMER, name=ome.services.pixeldata.PixelDataThread.batch, count=323, min=4.088312, max=20.670633, mean=12.3794725, stddev=11.725471626912093, median=12.3794725, p75=20.670633, p95=20.670633, p98=20.670633, p99=20.670633, p999=20.670633, mean_rate=2.193700041194688E-4, m1=5.778884146644216E-9, m5=1.7016008277662782E-4, m15=4.121759882047415E-4, rate_unit=events/second, duration_unit=milliseconds
2017-10-16 08:56:42,931 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 14041/140400 (10%).
2017-10-16 09:11:32,284 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 28081/140400 (20%).
2017-10-16 09:22:27,813 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 42121/140400 (30%).
2017-10-16 09:38:34,959 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 56161/140400 (40%).
2017-10-16 09:51:22,555 INFO  [                      ome.system.metrics] (r-thread-1) type=TIMER, name=ome.services.pixeldata.PixelDataThread.batch, count=323, min=4.088312, max=20.670633, mean=12.3794725, stddev=11.725471626912093, median=12.3794725, p75=20.670633, p95=20.670633, p98=20.670633, p99=20.670633, p999=20.670633, mean_rate=2.1883495468831373E-4, m1=5.060286122646805E-35, m5=1.0454996826395106E-9, m15=7.549266558565082E-6, rate_unit=events/second, duration_unit=milliseconds
2017-10-16 09:58:58,758 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 70201/140400 (50%).
2017-10-16 10:10:32,823 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 84241/140400 (60%).
2017-10-16 10:22:34,409 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 98281/140400 (70%).
2017-10-16 10:45:26,516 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 112321/140400 (80%).
2017-10-16 10:51:22,555 INFO  [                      ome.system.metrics] (r-thread-1) type=TIMER, name=ome.services.pixeldata.PixelDataThread.batch, count=323, min=4.088312, max=20.670633, mean=12.3794725, stddev=11.725471626912093, median=12.3794725, p75=20.670633, p95=20.670633, p98=20.670633, p99=20.670633, p999=20.670633, mean_rate=2.1830250888146442E-4, m1=4.431044989528208E-61, m5=6.423772065474417E-15, m15=1.3826964016147306E-7, rate_unit=events/second, duration_unit=milliseconds
2017-10-16 10:56:10,355 INFO  [                ome.io.nio.PixelsService] (2-thread-3) Pyramid creation for Pixels:812259 126361/140400 (90%).
2017-10-16 11:06:37,806 INFO  [                ome.io.nio.PixelsService] (2-thread-3) SUCCESS -- Pyramid created for pixels id:812259
2017-10-16 11:06:37,826 INFO  [       loci.formats.in.MinimalTiffReader] (2-thread-3) Reading IFDs
2017-10-16 11:06:37,843 INFO  [       loci.formats.in.MinimalTiffReader] (2-thread-3) Populating metadata
2017-10-16 11:06:38,237 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) Unknown JPEG 2000 box 0x290000 at 749256
2017-10-16 11:06:38,237 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) File is a raw codestream not a JP2.
2017-10-16 11:06:38,584 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) Unknown JPEG 2000 box 0x290000 at 2463322981
2017-10-16 11:06:38,585 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) File is a raw codestream not a JP2.
2017-10-16 11:06:38,595 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) Unknown JPEG 2000 box 0xc7f4f6a1 at 2451788470
2017-10-16 11:06:38,605 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) Unknown JPEG 2000 box 0x25a3d892 at 2109537927
2017-10-16 11:06:38,614 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) Unknown JPEG 2000 box 0x2deb3e64 at 382680608
2017-10-16 11:06:38,667 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) Unknown JPEG 2000 box 0x6781989a at 214293002
2017-10-16 11:06:38,963 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) Unknown JPEG 2000 box 0x290000 at 4925845350
2017-10-16 11:06:38,963 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) File is a raw codestream not a JP2.
2017-10-16 11:06:39,008 INFO  [  loci.formats.in.JPEG2000MetadataParser] (2-thread-3) Unknown JPEG 2000 box 0x76e5b130 at 4914310839
2017-10-16 11:06:39,009 INFO  [              loci.formats.in.TiffReader] (2-thread-3) Checking comment style
2017-10-16 11:06:39,009 INFO  [          loci.formats.in.BaseTiffReader] (2-thread-3) Populating OME metadata
2017-10-16 11:10:11,346 ERROR [        ome.services.util.ServiceHandler] (2-thread-3) java.lang.Error:  Wrapped Exception: (java.lang.OutOfMemoryError):
Requested array size exceeds VM limit
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
        at java.util.Arrays.copyOf(Arrays.java:2271) ~[na:1.7.0_151]
        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[na:1.7.0_151]
        at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[na:1.7.0_151]
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:135) ~[na:1.7.0_151]
        at java.io.DataOutputStream.writeByte(DataOutputStream.java:153) ~[na:1.7.0_151]
        at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:350) ~[formats-bsd.jar:5.5.3]
        at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:277) ~[formats-bsd.jar:5.5.3]
        at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:270) ~[formats-bsd.jar:5.5.3]
        at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:204) ~[formats-bsd.jar:5.5.3]
        at ome.io.bioformats.OmeroPixelsPyramidWriter.recompressSeries(OmeroPixelsPyramidWriter.java:127) ~[romio.jar:na]
        at ome.io.bioformats.OmeroPixelsPyramidWriter.postProcess(OmeroPixelsPyramidWriter.java:88) ~[romio.jar:na]
        at ome.io.bioformats.OmeroPixelsPyramidWriter.close(OmeroPixelsPyramidWriter.java:52) ~[romio.jar:na]
        at ome.io.bioformats.BfPyramidPixelBuffer.closeWriter(BfPyramidPixelBuffer.java:436) ~[romio.jar:na]
        at ome.io.bioformats.BfPyramidPixelBuffer.close(BfPyramidPixelBuffer.java:671) ~[romio.jar:na]
        at ome.io.nio.PixelsService.makePyramid(PixelsService.java:322) ~[romio.jar:na]
        at ome.services.pixeldata.PixelDataHandler.process(PixelDataHandler.java:144) ~[server.jar:na]
        at ome.services.pixeldata.PixelDataHandler.handleEventLog(PixelDataHandler.java:111) ~[server.jar:na]
        at ome.services.pixeldata.PixelDataThread$HandleEventLog.doWork(PixelDataThread.java:246) ~[server.jar:na]
        at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_151]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_151]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) ~[spring-aop.jar:3.0.1.RELEASE]
        at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:562) ~[server.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at ome.security.basic.NullEventHandler.invoke(NullEventHandler.java:39) ~[server.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111) ~[spring-orm.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108) ~[spring-tx.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
2017-10-16 11:10:11,353 WARN  [        ome.services.util.ServiceHandler] (2-thread-3) Unknown exception thrown.

java.lang.OutOfMemoryError: Requested array size exceeds VM limit
        at java.util.Arrays.copyOf(Arrays.java:2271) ~[na:1.7.0_151]
        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[na:1.7.0_151]
        at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[na:1.7.0_151]
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:135) ~[na:1.7.0_151]
        at java.io.DataOutputStream.writeByte(DataOutputStream.java:153) ~[na:1.7.0_151]
        at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:350) ~[formats-bsd.jar:5.5.3]
        at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:277) ~[formats-bsd.jar:5.5.3]
        at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:270) ~[formats-bsd.jar:5.5.3]
        at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:204) ~[formats-bsd.jar:5.5.3]
        at ome.io.bioformats.OmeroPixelsPyramidWriter.recompressSeries(OmeroPixelsPyramidWriter.java:127) ~[romio.jar:na]
        at ome.io.bioformats.OmeroPixelsPyramidWriter.postProcess(OmeroPixelsPyramidWriter.java:88) ~[romio.jar:na]
        at ome.io.bioformats.OmeroPixelsPyramidWriter.close(OmeroPixelsPyramidWriter.java:52) ~[romio.jar:na]
        at ome.io.bioformats.BfPyramidPixelBuffer.closeWriter(BfPyramidPixelBuffer.java:436) ~[romio.jar:na]
        at ome.io.bioformats.BfPyramidPixelBuffer.close(BfPyramidPixelBuffer.java:671) ~[romio.jar:na]
        at ome.io.nio.PixelsService.makePyramid(PixelsService.java:322) ~[romio.jar:na]
        at ome.services.pixeldata.PixelDataHandler.process(PixelDataHandler.java:144) ~[server.jar:na]
        at ome.services.pixeldata.PixelDataHandler.handleEventLog(PixelDataHandler.java:111) ~[server.jar:na]
        at ome.services.pixeldata.PixelDataThread$HandleEventLog.doWork(PixelDataThread.java:246) ~[server.jar:na]
        at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_151]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_151]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) ~[spring-aop.jar:3.0.1.RELEASE]
        at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:562) ~[server.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at ome.security.basic.NullEventHandler.invoke(NullEventHandler.java:39) ~[server.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111) ~[spring-orm.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108) ~[spring-tx.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
2017-10-16 11:10:11,354 ERROR [        ome.services.util.ServiceHandler] (2-thread-3) Method interface ome.services.util.Executor$Work.doWork invocation took 9223276
2017-10-16 11:10:11,354 ERROR [  ome.services.pixeldata.PixelDataThread] (2-thread-2) ExceptionException!
ome.conditions.InternalException:  Wrapped Exception: (java.lang.OutOfMemoryError):
Requested array size exceeds VM limit
        at java.util.Arrays.copyOf(Arrays.java:2271) ~[na:1.7.0_151]
        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[na:1.7.0_151]
        at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[na:1.7.0_151]
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:135) ~[na:1.7.0_151]
        at java.io.DataOutputStream.writeByte(DataOutputStream.java:153) ~[na:1.7.0_151]
        at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:350) ~[formats-bsd.jar:5.5.3]
        at loci.formats.tiff.TiffSaver.writeImage(TiffSaver.java:277) ~[formats-bsd.jar:5.5.3]
        at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:270) ~[formats-bsd.jar:5.5.3]
        at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:204) ~[formats-bsd.jar:5.5.3]
        at ome.io.bioformats.OmeroPixelsPyramidWriter.recompressSeries(OmeroPixelsPyramidWriter.java:127) ~[romio.jar:na]
        at ome.io.bioformats.OmeroPixelsPyramidWriter.postProcess(OmeroPixelsPyramidWriter.java:88) ~[romio.jar:na]
        at ome.io.bioformats.OmeroPixelsPyramidWriter.close(OmeroPixelsPyramidWriter.java:52) ~[romio.jar:na]
        at ome.io.bioformats.BfPyramidPixelBuffer.closeWriter(BfPyramidPixelBuffer.java:436) ~[romio.jar:na]
        at ome.io.bioformats.BfPyramidPixelBuffer.close(BfPyramidPixelBuffer.java:671) ~[romio.jar:na]
        at ome.io.nio.PixelsService.makePyramid(PixelsService.java:322) ~[romio.jar:na]
        at ome.services.pixeldata.PixelDataHandler.process(PixelDataHandler.java:144) ~[server.jar:na]
        at ome.services.pixeldata.PixelDataHandler.handleEventLog(PixelDataHandler.java:111) ~[server.jar:na]
        at ome.services.pixeldata.PixelDataThread$HandleEventLog.doWork(PixelDataThread.java:246) ~[server.jar:na]
        at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_151]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_151]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) ~[spring-aop.jar:3.0.1.RELEASE]
        at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:562) ~[server.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at ome.security.basic.NullEventHandler.invoke(NullEventHandler.java:39) ~[server.jar:na]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111) ~[spring-orm.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108) ~[spring-tx.jar:3.0.1.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop.jar:3.0.1.RELEASE]


My current jvmcfg settings are:
Code: Select all
omero admin jvmcfg
JVM Settings:
============
blitz=-Xmx38400m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '128000', 'percent': '30'})
indexer=-Xmx19200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '128000', 'percent': '15'})
pixeldata=-Xmx38400m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '128000', 'percent': '30'})
repository=-Xmx19200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '128000', 'percent': '15'})


I thought these were pretty high. Where should I increase memory to get the import to fully complete?

Thanks,
- Damir
dsudar
 
Posts: 235
Joined: Mon May 14, 2012 8:43 pm
Location: Berkeley, CA, USA

Re: pyramid creation on large image - out of memory

Postby mtbc » Tue Oct 17, 2017 11:02 am

Dear Damir,

It's indeed surprising that you're running into this issue. You are already doing the right thing in configuring the pixel data server process with plenty of memory; I don't know how much scope you have to try further increasing it nor if that extra would make the difference.

Is it always the same image of the set that triggers the problem? If so, would you be able to provide it to us so we can try some profiling at our end to see if there is some issue with TiffWriter or similar?

Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Re: pyramid creation on large image - out of memory

Postby dsudar » Tue Oct 17, 2017 12:36 pm

Hi Mark,
The image is quite large (5.5GB). Can you provide alternative upload instructions?
Thanks,
Damir
dsudar
 
Posts: 235
Joined: Mon May 14, 2012 8:43 pm
Location: Berkeley, CA, USA

Re: pyramid creation on large image - out of memory

Postby mlinkert » Tue Oct 17, 2017 4:21 pm

Hi Damir,

I will send FTP upload information via private message.

Regards,
-Melissa
User avatar
mlinkert
Team Member
 
Posts: 353
Joined: Fri May 29, 2009 2:12 pm
Location: Southwest Wisconsin

Re: pyramid creation on large image - out of memory

Postby mtbc » Wed Oct 18, 2017 2:12 pm

Damir,

Which version of OMERO are you running?

Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Re: pyramid creation on large image - out of memory

Postby dsudar » Wed Oct 18, 2017 3:09 pm

Hi Mark,
This is still on 5.3.5. I am planning upgrading to 5.4 in the next week or so.
Thanks,
Damir
dsudar
 
Posts: 235
Joined: Mon May 14, 2012 8:43 pm
Location: Berkeley, CA, USA

Re: pyramid creation on large image - out of memory

Postby mtbc » Thu Oct 19, 2017 12:35 pm

Dear Damir,

I can at least reproduce your issue with OMERO 5.4.0: I get the same out of memory error in OmeroPixelsPyramidWriter.recompressSeries after building the pyramid for your 115,063 × 106,010 image; I had pixeldata=-Xmx30367m so just a little lower than yours. I'm afraid that I'd thus expect that you'll still see this error after upgrading but, thanks to your image submission, at least we can investigate at our end.

Cheers,
Mark
Attachments
logs.tar.bz2
(1.79 MiB) Downloaded 184 times
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Re: pyramid creation on large image - out of memory

Postby dsudar » Thu Feb 22, 2018 5:56 am

Hi team,

I was wondering if there is any insight yet into the cause of this OOM issue. Meanwhile more folks at OHSU are creating files that have this same problem. Would it help to have a few more examples? I'll be happy to upload to the alternative upload site since these are large.

Cheers,
- Damir
dsudar
 
Posts: 235
Joined: Mon May 14, 2012 8:43 pm
Location: Berkeley, CA, USA

Re: pyramid creation on large image - out of memory

Postby jmoore » Thu Feb 22, 2018 12:50 pm

Hi Damir,

Mark reproduced with a trivial example (see https://trello.com/c/fFBOR0rY/282-tiffsaver-oom-on-big-image-import#comment-59e9ac132ed089f6d553a1b8) so sending more files won't be necessary (though as always, thanks for the offer!) We don't however have a proposed fix yet.

Do you have the system resources and, if so, have you tried upping the memory on the process?
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: pyramid creation on large image - out of memory

Postby dsudar » Thu Feb 22, 2018 5:55 pm

Hi Josh,

Thanks for pointing me at the Trello card, I had missed that.

I already have pretty high memory allocation:
Code: Select all
omero_user@omero:~$ omero admin jvmcfg
JVM Settings:
============
blitz=-Xmx38400m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '128000', 'percent': '30'})
indexer=-Xmx19200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '128000', 'percent': '15'})
pixeldata=-Xmx38400m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '128000', 'percent': '30'})
repository=-Xmx19200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '128000', 'percent': '15'})

but I do have some headroom and will try to go even higher. I presume both Blitz and Pixeldata should get more?

However, when reading the Trello card I see a note from @davidgault1 that there may be a hard limit array size of 2^31. If that's true, I guess increasing memory won't help, right?

One sneaky issue I noticed is: after the OOM failure, no new pyramid creation is even initiating until I restart the OMERO.server. I.e. nothing is being logged in PixelData-0.log. After a restart PixelData goes to work on the next pyramid. So that OOM does mess something up badly but silently.

Thanks,
- Damir
dsudar
 
Posts: 235
Joined: Mon May 14, 2012 8:43 pm
Location: Berkeley, CA, USA

Next

Return to User Discussion

Who is online

Users browsing this forum: No registered users and 1 guest