We're Hiring!

All Imports failing...

Having a problem deploying OMERO? Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

The OMERO.server installation documentation begins here and you can find OMERO.web deployment documentation here.

Re: All Imports failing...

Postby mtbc » Tue Sep 22, 2015 9:04 am

Dear Rick,

Unfortunately the most clueful people are now busy in meetings for the IDR project over the next couple of days: it is an unusually busy time for the OME team. We will make sure that somebody helps you this week, but we certainly regret the delay.

In the meantime, all I can suggest is to check your logs again and if you see some difference from before then go ahead and send us a new set. And, of course, if our delay in helping is too great, I'd alternatively suggest restoring from a backup taken before the problem started.

To make sure I am clear on this:
1. Your OMERO server ran into the system open-files limit during a large import.
2. No further imports could be made.
3. You increased the open-files limit substantially.
4. Despite restarts, deleting duplicate repositories, etc., your OMERO server is not recovering ability to accept further imports.

I wonder if anybody else in the community has run into the same issue of having to recover from this circumstance?

Cheers,

Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Re: All Imports failing...

Postby RickB » Tue Sep 22, 2015 12:50 pm

Thanks, yes, that sums it up. I appreciate the help...
RickB
 
Posts: 24
Joined: Sat Jun 27, 2015 11:34 am

Re: All Imports failing...

Postby jmoore » Thu Sep 24, 2015 11:58 am

Hi Rick,

sorry for not coming back sooner; you've certainly found an interesting one for us.

Could you possible send the output of:

Code: Select all
jstack $(bin/omero admin ice server pid Blitz-0)


Cheers,
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: All Imports failing...

Postby RickB » Thu Sep 24, 2015 12:54 pm

No problem, here is the output:


2015-09-24 08:51:48
Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.7-b01 mixed mode):

"Blitz-0-Ice.ThreadPool.Server-1131" prio=10 tid=0x00007f8350021800 nid=0x3cfe in Object.wait() [0x00007f83c39f8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007f72078b8> (a IceInternal.ThreadPool)
at IceInternal.ThreadPool.followerWait(ThreadPool.java:554)
- eliminated <0x00000007f72078b8> (a IceInternal.ThreadPool)
at IceInternal.ThreadPool.run(ThreadPool.java:375)
- locked <0x00000007f72078b8> (a IceInternal.ThreadPool)
at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:643)
at java.lang.Thread.run(Thread.java:722)

"Attach Listener" daemon prio=10 tid=0x00007f83a4001000 nid=0x3cde waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Blitz-0-Ice.ThreadPool.Client-336" prio=10 tid=0x00007f835003a000 nid=0x3ca6 in Object.wait() [0x00007f83c8cbb000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007f6de0000> (a IceInternal.ThreadPool)
at IceInternal.ThreadPool.followerWait(ThreadPool.java:554)
- eliminated <0x00000007f6de0000> (a IceInternal.ThreadPool)
at IceInternal.ThreadPool.run(ThreadPool.java:375)
- locked <0x00000007f6de0000> (a IceInternal.ThreadPool)
at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:643)
at java.lang.Thread.run(Thread.java:722)

"Blitz-0-Ice.ThreadPool.Client-334" prio=10 tid=0x00007f8350005000 nid=0x3c3f runnable [0x00007f83c90bf000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000007f6de0440> (a sun.nio.ch.Util$2)
- locked <0x00000007f6de0430> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007f6de01c0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at IceInternal.Selector.select(Selector.java:262)
at IceInternal.ThreadPool.run(ThreadPool.java:319)
at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:643)
at java.lang.Thread.run(Thread.java:722)

"Blitz-0-Ice.ThreadPool.Server-1105" prio=10 tid=0x00007f8350033000 nid=0x3b94 runnable [0x00007f83c8fbe000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000007f7207aa0> (a sun.nio.ch.Util$2)
- locked <0x00000007f7213fc8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007f7206300> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at IceInternal.Selector.select(Selector.java:262)
at IceInternal.ThreadPool.run(ThreadPool.java:319)
at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:643)
at java.lang.Thread.run(Thread.java:722)

"pool-2-thread-5" prio=10 tid=0x00007f8364009800 nid=0xb67 waiting on condition [0x00007f83c82b1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007b5e30058> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

"pool-2-thread-4" prio=10 tid=0x00007f8364007800 nid=0xb51 runnable [0x00007f83c84b1000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:89)
at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:982)
at java.nio.channels.FileChannel.lock(FileChannel.java:1052)
at ome.services.blitz.repo.FileMaker.getLine(FileMaker.java:95)
- locked <0x00000007f7214b60> (a [Ljava.lang.Object;)
at ome.services.blitz.repo.AbstractRepositoryI$GetOrCreateRepo.doWork(AbstractRepositoryI.java:340)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:566)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy72.doWork(Unknown Source)
at ome.services.util.Executor$Impl.execute(Executor.java:447)
at ome.services.util.Executor$Impl.execute(Executor.java:391)
at ome.services.blitz.repo.AbstractRepositoryI.takeover(AbstractRepositoryI.java:210)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)
at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:264)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

"pool-2-thread-3" prio=10 tid=0x00007f8364005800 nid=0xb50 runnable [0x00007f83c85b2000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:89)
at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:982)
at java.nio.channels.FileChannel.lock(FileChannel.java:1052)
at ome.services.blitz.repo.FileMaker.getLine(FileMaker.java:95)
- locked <0x00000007f718ccd0> (a [Ljava.lang.Object;)
at ome.services.blitz.repo.AbstractRepositoryI$GetOrCreateRepo.doWork(AbstractRepositoryI.java:340)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:566)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy72.doWork(Unknown Source)
at ome.services.util.Executor$Impl.execute(Executor.java:447)
at ome.services.util.Executor$Impl.execute(Executor.java:391)
at ome.services.blitz.repo.AbstractRepositoryI.takeover(AbstractRepositoryI.java:210)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)
at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:264)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

"pool-2-thread-2" prio=10 tid=0x00007f8364004800 nid=0xb4f waiting on condition [0x00007f83c86b5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007b5e30058> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

"Blitz-0-Ice.EndpointHostResolverThread" prio=10 tid=0x00007f83d4504800 nid=0xb37 in Object.wait() [0x00007f83c93c2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007f72144e8> (a IceInternal.EndpointHostResolver)
at java.lang.Object.wait(Object.java:503)
at IceInternal.EndpointHostResolver.run(EndpointHostResolver.java:89)
- locked <0x00000007f72144e8> (a IceInternal.EndpointHostResolver)
at IceInternal.EndpointHostResolver$HelperThread.run(EndpointHostResolver.java:152)

"Blitz-0-Ice.Timer" prio=10 tid=0x00007f83d4d6e000 nid=0xb36 in Object.wait() [0x00007f83c94c3000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007f6de5560> (a IceInternal.Timer)
at IceInternal.Timer.run(Timer.java:211)
- locked <0x00000007f6de5560> (a IceInternal.Timer)

"pool-2-thread-1" prio=10 tid=0x00007f8364001800 nid=0xb2b waiting on condition [0x00007f83c95c4000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007b5e30058> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

"OMERO.scheduler_QuartzSchedulerThread" prio=10 tid=0x00007f83d4d13800 nid=0xb29 waiting on condition [0x00007f83c9fde000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272)

"Ice.ThreadPool.Client-0" prio=10 tid=0x00007f83d4c78000 nid=0xb1e runnable [0x00007f83c9de5000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000007b2f6ffc8> (a sun.nio.ch.Util$2)
- locked <0x00000007b2f6ffb8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007b2f6fd68> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at IceInternal.Selector.select(Selector.java:262)
at IceInternal.ThreadPool.run(ThreadPool.java:319)
at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:643)
at java.lang.Thread.run(Thread.java:722)

"Ice.EndpointHostResolverThread" prio=10 tid=0x00007f83d4bf4800 nid=0xb1d in Object.wait() [0x00007f83ca0df000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007b2f72690> (a IceInternal.EndpointHostResolver)
at java.lang.Object.wait(Object.java:503)
at IceInternal.EndpointHostResolver.run(EndpointHostResolver.java:89)
- locked <0x00000007b2f72690> (a IceInternal.EndpointHostResolver)
at IceInternal.EndpointHostResolver$HelperThread.run(EndpointHostResolver.java:152)

"Ice.Timer" prio=10 tid=0x00007f83d4bd1000 nid=0xb1c in Object.wait() [0x00007f83ca1e0000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007b2f72920> (a IceInternal.Timer)
at IceInternal.Timer.run(Timer.java:211)
- locked <0x00000007b2f72920> (a IceInternal.Timer)

"PostgreSQL-JDBC-SharedTimer-1" daemon prio=10 tid=0x00007f83d4a51800 nid=0xb1b in Object.wait() [0x00007f83ca4e7000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007b2ec6cf8> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000007b2ec6cf8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"bitronix-task-scheduler" daemon prio=10 tid=0x00007f83d48b1800 nid=0xb19 waiting on condition [0x00007f83ca5e8000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at bitronix.tm.timer.TaskScheduler.run(TaskScheduler.java:245)

"net.sf.ehcache.CacheManager@64e6bd1e" daemon prio=10 tid=0x00007f83d49ce800 nid=0xb18 in Object.wait() [0x00007f83ca6e9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007b2e9e960> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000007b2e9e960> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"perf4j-async-stats-appender-sink-CoalescingStatistics" daemon prio=10 tid=0x00007f83d469c000 nid=0xaaf waiting on condition [0x00007f83ca7ea000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007b2a52908> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
at org.perf4j.helpers.GenericAsyncCoalescingStatisticsAppender$StopWatchesFromQueueIterator.getNext(GenericAsyncCoalescingStatisticsAppender.java:388)
at org.perf4j.helpers.GenericAsyncCoalescingStatisticsAppender$StopWatchesFromQueueIterator.hasNext(GenericAsyncCoalescingStatisticsAppender.java:349)
at org.perf4j.helpers.GroupingStatisticsIterator.getNext(GroupingStatisticsIterator.java:149)
at org.perf4j.helpers.GroupingStatisticsIterator.hasNext(GroupingStatisticsIterator.java:102)
at org.perf4j.helpers.GenericAsyncCoalescingStatisticsAppender$Dispatcher.run(GenericAsyncCoalescingStatisticsAppender.java:314)
at java.lang.Thread.run(Thread.java:722)

"Service Thread" daemon prio=10 tid=0x00007f83d40f1800 nid=0xa64 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x00007f83d40ef000 nid=0xa63 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x00007f83d40ec000 nid=0xa62 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00007f83d40ea000 nid=0xa61 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00007f83d409c800 nid=0xa60 in Object.wait() [0x00007f83cbcfb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007b2a390f0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000007b2a390f0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)

"Reference Handler" daemon prio=10 tid=0x00007f83d409a800 nid=0xa5f in Object.wait() [0x00007f83cbdfc000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007b2a38c30> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00000007b2a38c30> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x00007f83d4009800 nid=0xa59 in Object.wait() [0x00007f83da95f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007f7205a88> (a IceInternal.ObjectAdapterFactory)
at java.lang.Object.wait(Object.java:503)
at IceInternal.ObjectAdapterFactory.waitForShutdown(ObjectAdapterFactory.java:60)
- locked <0x00000007f7205a88> (a IceInternal.ObjectAdapterFactory)
at Ice.CommunicatorI.waitForShutdown(CommunicatorI.java:29)
at ome.services.blitz.Entry.start(Entry.java:207)
at ome.services.blitz.Entry.main(Entry.java:151)

"VM Thread" prio=10 tid=0x00007f83d4093000 nid=0xa5e runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f83d4017000 nid=0xa5a runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f83d4019000 nid=0xa5b runnable

"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f83d401a800 nid=0xa5c runnable

"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f83d401c800 nid=0xa5d runnable

"VM Periodic Task Thread" prio=10 tid=0x00007f83d4104000 nid=0xa65 waiting on condition

JNI global references: 194
RickB
 
Posts: 24
Joined: Sat Jun 27, 2015 11:34 am

Re: All Imports failing...

Postby jmoore » Fri Sep 25, 2015 8:27 am

Thanks, Rick.

Code: Select all
at ome.services.blitz.repo.FileMaker.getLine(FileMaker.java:95)
- locked <0x00000007f718ccd0> (a [Ljava.lang.Object;)


points to the fact that your server is still trying to get a lock file on the directories:


You mentioned having moved the .omero directories and symlinking them back. Can you show us a file listing? Did something get overwritten, perhaps? cF

Code: Select all
ls -ltrad . .omero .omero/* ManagedRepository ManagedRepository/.omero ManagedRepository/.omero/*
drwxrwxr-x   3 a_user a_user  4096 May 30  2014 .omero
drwxrwxr-x   3 a_user a_user  4096 Sep 25 04:58 ManagedRepository/.omero/repository
drwxrwxr-x   3 a_user a_user  4096 Sep 25 04:58 ManagedRepository/.omero
drwxrwxr-x 468 a_user a_user 32768 Sep 25 04:58 .omero/repository
drwxr-xr-x  11 a_user a_user  4096 Sep 25 05:05 .
drwxrwxr-x 216 a_user a_user 20480 Sep 25 06:44 ManagedRepository


Cheers,
~Josh

P.S. you also mentioned NFS v3. Some feedback we've received points to the locking in v4 being better than in v3.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: All Imports failing...

Postby RickB » Fri Sep 25, 2015 1:51 pm

Thanks Josh, I was able to import successfully after moving both the /OMERO/.omero as well as /OMERO/Managedrepository/.omero to local disk and symlinking, as well as removing old lockfiles ( I had missed one in my previous test). We've always had our /OMERO repository on an NFS share via NFS 3, strange that it broke so suddenly. I'm looking at our NAS to see if anything has changed on that side as far as locking. Thanks very much for the help, we'll look at NFS v4 or potentially SAN/local disk for the repo moving forward.
Rick
RickB
 
Posts: 24
Joined: Sat Jun 27, 2015 11:34 am

Re: All Imports failing...

Postby mtbc » Fri Sep 25, 2015 2:10 pm

Dear Rick,

Thank goodness, we are most pleased to hear of your progress! Thank you for providing all the useful information along the way. It will be interesting to find out if there was some remote storage change coinciding with your issues.

Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Previous

Return to Installation and Deployment

Who is online

Users browsing this forum: No registered users and 1 guest