We're Hiring!

OMERO Bulk import

General user discussion about using the OMERO platform to its fullest. Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

Re: OMERO Bulk import

Postby alexr » Fri Jun 22, 2018 12:03 pm

Looks to me as if it should be fine:

Code: Select all
>>> print(connIP)
130.83.98.66
>>> type(connIP)
<type 'str'>


:shock:
Alex
alexr
 
Posts: 46
Joined: Tue Jun 12, 2018 12:20 pm

Re: OMERO Bulk import

Postby alexr » Tue Jun 26, 2018 12:49 pm

Hi Josh,
somehow I could solve the problem with the variables. Now I have an other issue.
If I start an import using the above mentioned commands using the bulk.yaml files and the filelist.tsv. Is is possible that the importer gets stuck and if so how can I check for that.
During testing I imported successfully >10 folders with small samples of images. Then I tried to import a large Volocity (Perkin Elmer) library. These libraries give me sometimes trouble, also when I import them using the insight client.
The Dataset of the Volocity Import shows in the Omero web and insight client, but no previews are displayed. When I try to open these I get a message displayed that the preview needs to be calculated.
Then after a long time the preview is visible.
Is it possible that during this time new imports are started using the bulk/filelist method? Or is the server busy during this time and this is not possible. Because I realised that during this time me and other users do not get the normal stdout from the import command.
Is there a way to check the activities on the OMERO server that are currently running?
Thanks Alex
alexr
 
Posts: 46
Joined: Tue Jun 12, 2018 12:20 pm

Re: OMERO Bulk import

Postby alexr » Tue Jun 26, 2018 1:34 pm

Hi Josh,
a small update. I checked the Managed Repositories on the OMERO Server. After the Volocity file was imported all other imports seam to have failed. I get the entries for the time stampes such as:
/OMERO/ManagedRepository/Alex R_3/2018-06/26/14-57-13.868
but instead of having the soft link in there it is empty. The same applies for all users.
I have upploaded the latest log life in the ManagedRepository since I hope you can get some clue from this. (http://qa.openmicroscopy.org.uk/qa/upload/)
Restarting OMEROweb or OMERO server complete dose not help.
But import via insight still works.
Are there any possible locks?
Thanks Alex
alexr
 
Posts: 46
Joined: Tue Jun 12, 2018 12:20 pm

Re: OMERO Bulk import

Postby jmoore » Tue Jun 26, 2018 2:15 pm

alexr wrote:If I start an import using the above mentioned commands using the bulk.yaml files and the filelist.tsv. Is is possible that the importer gets stuck and if so how can I check for that.


If you have the JDK installed, then `jps` should list all the Java processes running on the system. You can either use `jstack` to list what that process is doing or if you don't have that installed, `kill -QUIT` with the process ID will print out a stack trace to stdout that you can send us.

During testing I imported successfully >10 folders with small samples of images. Then I tried to import a large Volocity (Perkin Elmer) library. These libraries give me sometimes trouble, also when I import them using the insight client.
The Dataset of the Volocity Import shows in the Omero web and insight client, but no previews are displayed. When I try to open these I get a message displayed that the preview needs to be calculated.
Then after a long time the preview is visible.


It certainly sounds possible that this particular import could take a while.

Is it possible that during this time new imports are started using the bulk/filelist method? Or is the server busy during this time and this is not possible. Because I realised that during this time me and other users do not get the normal stdout from the import command.


Hmmm.... I'm a bit unclear on what's going on. Can you show us examples of what you're seeing?


Is there a way to check the activities on the OMERO server that are currently running?



`bin/omero sessions who` will show you who is logged in. Otherwise, you can use `jstack` with the server's pid.


alexr wrote:I checked the Managed Repositories on the OMERO Server. After the Volocity file was imported all other imports seam to have failed. I get the entries for the time stampes such as:
/OMERO/ManagedRepository/Alex R_3/2018-06/26/14-57-13.868
but instead of having the soft link in there it is empty. The same applies for all users.
I have upploaded the latest log life in the ManagedRepository since I hope you can get some clue from this.


The log doesn't show much other than that it's a sizeable import. Can you send the server log as well?

Are there any possible locks?


Not that I can think of.

~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: OMERO Bulk import

Postby alexr » Tue Jun 26, 2018 2:51 pm

Hi Josh,
thanks for the answer. I don't know if I get it right. If I use jps I get:
Code: Select all
~$ jps -l
25656 sun.tools.jps.Jps

alternatively
Code: Select all
ps -aef | grep java
omerous+ 23030 22997  2 15:23 ?        00:02:06 java -Xmx2463m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions -Djava.awt.headless=true -Dlogback.configurationFile=etc/logback.xml -Domero.logfile=var/log/${omero.name}.log -Domero.name=Blitz-0 ome.services.blitz.Entry --Ice.Config=/home/omerouser/OMERO.server-5.4.5-ice36-b83/var/master/servers/Blitz-0/config/config
omerous+ 23060 22997  1 15:23 ?        00:00:52 java -Xmx2463m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions -Djava.awt.headless=true -Dlogback.configurationFile=etc/logback-indexing.xml -Domero.logfile=var/log/${omero.name}.log -Domero.name=PixelData-0 -jar lib/server/blitz.jar ome.pixeldata --Ice.Config=/home/omerouser/OMERO.server-5.4.5-ice36-b83/var/master/servers/PixelData-0/config/config
omerous+ 25588 22997 15 16:35 ?        00:00:26 java -Xmx1642m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions -Djava.awt.headless=true -Dlogback.configurationFile=etc/logback-indexing.xml -Domero.logfile=var/log/${omero.name}.log -Domero.name=Indexer-0 -jar lib/server/blitz.jar ome.fulltext --Ice.Config=/home/omerouser/OMERO.server-5.4.5-ice36-b83/var/master/servers/Indexer-0/config/config
omero    25719 25027  0 16:38 pts/8    00:00:00 grep --color=auto java


When I try to run jstack I get:
Code: Select all
sudo jstack 23030
23030: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

Then I tried the -F option and get
Code: Select all
Attaching to process ID 22997, please wait...
sun.jvm.hotspot.debugger.NoSuchSymbolException: Could not find symbol "gHotSpotVMTypes" in any of the known library names (libjvm.so, libjvm_g.so, gamma_g)
   at sun.jvm.hotspot.HotSpotTypeDataBase.lookupInProcess(HotSpotTypeDataBase.java:585)
   at sun.jvm.hotspot.HotSpotTypeDataBase.readVMTypes(HotSpotTypeDataBase.java:150)
   at sun.jvm.hotspot.HotSpotTypeDataBase.<init>(HotSpotTypeDataBase.java:85)
   at sun.jvm.hotspot.bugspot.BugSpotAgent.setupVM(BugSpotAgent.java:569)
   at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:493)
   at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:331)
   at sun.jvm.hotspot.tools.Tool.start(Tool.java:163)
   at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)
   at sun.tools.jstack.JStack.main(JStack.java:102)
Debugger attached successfully.
jstack requires a java VM process/core!


I also checked the user on the OMERO system with
bin/omero sessions who
and get:
Code: Select all
Using session f7694a3d-14b8-4da3-984c-e70d39f4fea1 (Alex R@localhost:4064). Idle timeout: 10 min. Current group: AG_Cardoso
name   | group      | logged in               | agent          | timeout | hitCount | lastAccessTime     
--------+------------+-------------------------+----------------+---------+----------+---------------------
Alex R | AG_Cardoso | 2018-06-26 16:23:49     | OMERO.web      | 10 min. | 252      | 2018-06-26 16:24:54
root   | system     | 2018-06-26 15:24:26     | Python service | 10 min. | 1545     | 2018-06-26 16:26:08
guest  | guest      | 2018-06-26 16:23:49     | OMERO.web      | 10 min. | 18       | 2018-06-26 16:23:49
Alex R | AG_Cardoso | 2018-06-26 16:22:40 (*) | OMERO.sessions | 10 min. | 99       | 2018-06-26 16:26:30
root   | system     | 2018-06-26 15:24:26     | Python service | 10 min. | 100      | 2018-06-26 16:26:08


Although I (Alex R) am not connected by web or insight client.

And here is the ls of the ManagedRepository from today:
Code: Select all
drwxr-sr-x  2 omerouser inplace 4,0K Jun 26 13:46 13-46-21.420
-rw-r--r--  1 omerouser inplace  20K Jun 26 13:46 13-46-21.420.log
drwxr-sr-x  2 omerouser inplace 4,0K Jun 26 13:46 13-46-33.877
-rw-r--r--  1 omerouser inplace  20K Jun 26 13:46 13-46-33.877.log
drwxr-sr-x  2 omerouser inplace 4,0K Jun 26 14:56 14-56-45.702
drwxr-sr-x  2 omerouser inplace 4,0K Jun 26 14:56 14-56-55.094
drwxr-sr-x  2 omerouser inplace 4,0K Jun 26 14:57 14-57-04.448
drwxr-sr-x  2 omerouser inplace 4,0K Jun 26 14:57 14-57-13.868
drwxr-sr-x  2 omerouser inplace 4,0K Jun 26 14:57 14-57-23.329
drwxr-sr-x  2 omerouser inplace 4,0K Jun 26 14:57 14-57-32.572
drwxr-sr-x  2 omerouser inplace 4,0K Jun 26 14:57 14-57-41.950
drwxr-sr-x  2 omerouser inplace 4,0K Jun 26 14:57 14-57-51.284
drwxr-sr-x  2 omerouser inplace 4,0K Jun 26 14:58 14-58-00.670
drwxr-sr-x  2 omerouser inplace 4,0K Jun 26 14:58 14-58-10.326

The 13-46-33.877 is the last successful import, all the following imports are empty.

Thanks for the support,
Alex
alexr
 
Posts: 46
Joined: Tue Jun 12, 2018 12:20 pm

Re: OMERO Bulk import

Postby jmoore » Tue Jun 26, 2018 3:53 pm

Hi Alex,

alexr wrote:I don't know if I get it right. If I use jps I get:
Code: Select all
~$ jps -l
25656 sun.tools.jps.Jps



Are you currently logged in as the user that would be running Java?


alternatively
Code: Select all
ps -aef | grep java



None of these are imports, so it looks like only the Java servers are running.

When I try to run jstack I get:
Code: Select all
sudo jstack 23030
23030: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

Then I tried the -F option and get
Code: Select all
Attaching to process ID 22997, please wait...



You switched from 23030 to 22997. Try 23030 with the -F option.

I also checked the user on the OMERO system with
bin/omero sessions who
Code: Select all
Alex R | AG_Cardoso | 2018-06-26 16:23:49     | OMERO.web      | 10 min. | 252      | 2018-06-26 16:24:54
Alex R | AG_Cardoso | 2018-06-26 16:22:40 (*) | OMERO.sessions | 10 min. | 99       | 2018-06-26 16:26:30


Although I (Alex R) am not connected by web or insight client.


Not that it's overly important, but it looks like you might have a tab open somewhere.

~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: OMERO Bulk import

Postby alexr » Tue Jun 26, 2018 4:22 pm

Hi Josh,
sorry for the mistake, I was running the previous diagnostics as system administrator. Here are the results from the omero user:

Here is the jps -l result:
Code: Select all
23060 lib/server/blitz.jar
25588 lib/server/blitz.jar
26450 sun.tools.jps.Jps
23030 ome.services.blitz.Entry


Here are the three outputs for the 23060, 25588 and 23030, the 26450 does not give a result.
Code: Select all
omerous+ 23030     1  1 15:23 ?        00:02:23 java -Xmx2463m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions -Djava.awt.headless=true -Dlogback.configurationFile=etc/logback.xml -Domero.logfile=var/log/${omero.name}.log -Domero.name=Blitz-0 ome.services.blitz.Entry --Ice.Config=/home/omerouser/OMERO.server-5.4.5-ice36-b83/var/master/servers/Blitz-0/config/config
omerous+ 23060     1  0 15:23 ?        00:01:12 java -Xmx2463m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions -Djava.awt.headless=true -Dlogback.configurationFile=etc/logback-indexing.xml -Domero.logfile=var/log/${omero.name}.log -Domero.name=PixelData-0 -jar lib/server/blitz.jar ome.pixeldata --Ice.Config=/home/omerouser/OMERO.server-5.4.5-ice36-b83/var/master/servers/PixelData-0/config/config
omerous+ 25588     1  1 16:35 ?        00:01:15 java -Xmx1642m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions -Djava.awt.headless=true -Dlogback.configurationFile=etc/logback-indexing.xml -Domero.logfile=var/log/${omero.name}.log -Domero.name=Indexer-0 -jar lib/server/blitz.jar ome.fulltext --Ice.Config=/home/omerouser/OMERO.server-5.4.5-ice36-b83/var/master/servers/Indexer-0/config/config
omerous+ 26465 26425  0 18:12 pts/8    00:00:00 grep --color=auto java

jstack 23060
2018-06-26 18:13:29
Full thread dump OpenJDK 64-Bit Server VM (25.171-b11 mixed mode):

"Attach Listener" #26 daemon prio=9 os_prio=0 tid=0x00007fe428001000 nid=0x6770 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"pool-2-thread-5" #25 prio=5 os_prio=0 tid=0x00007fe400009000 nid=0x5b27 waiting on condition [0x00007fe3ebafb000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00000007316931a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-2-thread-4" #24 prio=5 os_prio=0 tid=0x00007fe400007800 nid=0x5b06 waiting on condition [0x00007fe3ebbfc000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00000007316931a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-2-thread-3" #23 prio=5 os_prio=0 tid=0x00007fe400005800 nid=0x5af0 waiting on condition [0x00007fe3ebcfd000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00000007316931a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"PixelData-0-Ice.Timer" #22 prio=5 os_prio=0 tid=0x00007fe461445000 nid=0x5aed waiting on condition [0x00007fe3ebffe000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00000007abe96208> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"PixelData-0-Ice.ThreadPool.Server-0" #21 prio=5 os_prio=0 tid=0x00007fe461414800 nid=0x5aec runnable [0x00007fe41c120000]
   java.lang.Thread.State: RUNNABLE
   at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
   at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
   at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   - locked <0x00000007abeb93c0> (a sun.nio.ch.Util$3)
   - locked <0x00000007abeb93b0> (a java.util.Collections$UnmodifiableSet)
   - locked <0x00000007abeb8ac0> (a sun.nio.ch.EPollSelectorImpl)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   at IceInternal.Selector.select(Selector.java:201)
   at IceInternal.ThreadPool.run(ThreadPool.java:414)
   at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
   at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
   at java.lang.Thread.run(Thread.java:748)

"PixelData-0-Ice.ThreadPool.Client-0" #20 prio=5 os_prio=0 tid=0x00007fe461408800 nid=0x5aeb runnable [0x00007fe41c221000]
   java.lang.Thread.State: RUNNABLE
   at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
   at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
   at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   - locked <0x00000007abe94798> (a sun.nio.ch.Util$3)
   - locked <0x00000007abe94788> (a java.util.Collections$UnmodifiableSet)
   - locked <0x00000007abe94550> (a sun.nio.ch.EPollSelectorImpl)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   at IceInternal.Selector.select(Selector.java:201)
   at IceInternal.ThreadPool.run(ThreadPool.java:414)
   at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
   at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
   at java.lang.Thread.run(Thread.java:748)

"pool-2-thread-2" #19 prio=5 os_prio=0 tid=0x00007fe400004800 nid=0x5aea waiting on condition [0x00007fe41c322000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00000007316931a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-2-thread-1" #18 prio=5 os_prio=0 tid=0x00007fe400001800 nid=0x5ae9 waiting on condition [0x00007fe41c423000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00000007316931a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"OMERO.scheduler_QuartzSchedulerThread" #17 prio=5 os_prio=0 tid=0x00007fe461145800 nid=0x5ae2 waiting on condition [0x00007fe41c724000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272)

"metrics-logger-reporter-thread-1" #15 daemon prio=5 os_prio=0 tid=0x00007fe460fce000 nid=0x5adc waiting on condition [0x00007fe4341ed000]
   java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00000007abdcd020> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"Ice.ThreadPool.Client-0" #14 prio=5 os_prio=0 tid=0x00007fe460b0b000 nid=0x5ab5 runnable [0x00007fe435b2c000]
   java.lang.Thread.State: RUNNABLE
   at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
   at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
   at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   - locked <0x0000000726a0e138> (a sun.nio.ch.Util$3)
   - locked <0x0000000726a0e128> (a java.util.Collections$UnmodifiableSet)
   - locked <0x0000000726a0dff0> (a sun.nio.ch.EPollSelectorImpl)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   at IceInternal.Selector.select(Selector.java:201)
   at IceInternal.ThreadPool.run(ThreadPool.java:414)
   at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
   at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
   at java.lang.Thread.run(Thread.java:748)

"net.sf.ehcache.CacheManager@5a18cd76" #12 daemon prio=5 os_prio=0 tid=0x00007fe460cf1000 nid=0x5ab1 in Object.wait() [0x00007fe435c2d000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000000726038b30> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:552)
   - locked <0x0000000726038b30> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:505)

"PostgreSQL-JDBC-SharedTimer-1" #11 daemon prio=5 os_prio=0 tid=0x00007fe46051a000 nid=0x5aae in Object.wait() [0x00007fe435f2e000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x000000072603ace0> (a java.util.TaskQueue)
   at java.lang.Object.wait(Object.java:502)
   at java.util.TimerThread.mainLoop(Timer.java:526)
   - locked <0x000000072603ace0> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:505)

"bitronix-task-scheduler" #10 daemon prio=5 os_prio=0 tid=0x00007fe460506800 nid=0x5aab waiting on condition [0x00007fe43622f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at bitronix.tm.timer.TaskScheduler.run(TaskScheduler.java:245)

"perf4j-async-stats-appender-sink-CoalescingStatistics" #9 daemon prio=5 os_prio=0 tid=0x00007fe4608b7800 nid=0x5aa0 waiting on condition [0x00007fe436948000]
   java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00000007260bb898> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
   at org.perf4j.helpers.GenericAsyncCoalescingStatisticsAppender$StopWatchesFromQueueIterator.getNext(GenericAsyncCoalescingStatisticsAppender.java:388)
   at org.perf4j.helpers.GenericAsyncCoalescingStatisticsAppender$StopWatchesFromQueueIterator.hasNext(GenericAsyncCoalescingStatisticsAppender.java:349)
   at org.perf4j.helpers.GroupingStatisticsIterator.getNext(GroupingStatisticsIterator.java:149)
   at org.perf4j.helpers.GroupingStatisticsIterator.hasNext(GroupingStatisticsIterator.java:102)
   at org.perf4j.helpers.GenericAsyncCoalescingStatisticsAppender$Dispatcher.run(GenericAsyncCoalescingStatisticsAppender.java:314)
   at java.lang.Thread.run(Thread.java:748)

"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007fe460176000 nid=0x5a33 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fe460161800 nid=0x5a32 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fe46015e800 nid=0x5a31 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fe46015d000 nid=0x5a30 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fe460129800 nid=0x5a27 in Object.wait() [0x00007fe464187000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000000726119c38> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
   - locked <0x0000000726119c38> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
   at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:212)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fe460125000 nid=0x5a25 in Object.wait() [0x00007fe464288000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000000726119ec0> (a java.lang.ref.Reference$Lock)
   at java.lang.Object.wait(Object.java:502)
   at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
   - locked <0x0000000726119ec0> (a java.lang.ref.Reference$Lock)
   at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=0 tid=0x00007fe46000b000 nid=0x5a18 in Object.wait() [0x00007fe46a406000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00000007abeb8270> (a IceInternal.ObjectAdapterFactory)
   at java.lang.Object.wait(Object.java:502)
   at IceInternal.ObjectAdapterFactory.waitForShutdown(ObjectAdapterFactory.java:63)
   - locked <0x00000007abeb8270> (a IceInternal.ObjectAdapterFactory)
   at Ice.CommunicatorI.waitForShutdown(CommunicatorI.java:32)
   at ome.services.blitz.Entry.start(Entry.java:202)
   at ome.services.blitz.Entry.main(Entry.java:146)

"VM Thread" os_prio=0 tid=0x00007fe46011d000 nid=0x5a24 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fe460020000 nid=0x5a1c runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fe460021800 nid=0x5a1e runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fe460023800 nid=0x5a1f runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007fe460179000 nid=0x5a35 waiting on condition

JNI global references: 228


jstack 23030
2018-06-26 18:14:06
Full thread dump OpenJDK 64-Bit Server VM (25.171-b11 mixed mode):

"Attach Listener" #281 daemon prio=9 os_prio=0 tid=0x00007f1a2c002000 nid=0x6782 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Blitz-0-Ice.ThreadPool.Client-76" #280 prio=5 os_prio=0 tid=0x00007f1a100d4000 nid=0x6706 runnable [0x00007f19e9bf1000]
   java.lang.Thread.State: RUNNABLE
   at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
   at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
   at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   - locked <0x000000072b5baad0> (a sun.nio.ch.Util$3)
   - locked <0x000000072bc2a718> (a java.util.Collections$UnmodifiableSet)
   - locked <0x000000072b5ba300> (a sun.nio.ch.EPollSelectorImpl)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   at IceInternal.Selector.select(Selector.java:201)
   at IceInternal.ThreadPool.run(ThreadPool.java:414)
   at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
   at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
   at java.lang.Thread.run(Thread.java:748)

"Blitz-0-Ice.ThreadPool.Server-146" #279 prio=5 os_prio=0 tid=0x00007f1a2844f800 nid=0x66fe in Object.wait() [0x00007f19e9cf2000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at IceInternal.ThreadPool.followerWait(ThreadPool.java:709)
   - locked <0x000000072b693d20> (a IceInternal.ThreadPool)
   at IceInternal.ThreadPool.run(ThreadPool.java:455)
   - locked <0x000000072b693d20> (a IceInternal.ThreadPool)
   at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
   at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
   at java.lang.Thread.run(Thread.java:748)

"Blitz-0-Ice.ThreadPool.Server-143" #275 prio=5 os_prio=0 tid=0x00007f19ec021800 nid=0x66f1 runnable [0x00007f1a06aee000]
   java.lang.Thread.State: RUNNABLE
   at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
   at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
   at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   - locked <0x000000072b693df8> (a sun.nio.ch.Util$3)
   - locked <0x000000072b69bf98> (a java.util.Collections$UnmodifiableSet)
   - locked <0x000000072b5bc9e8> (a sun.nio.ch.EPollSelectorImpl)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   at IceInternal.Selector.select(Selector.java:201)
   at IceInternal.ThreadPool.run(ThreadPool.java:414)
   at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
   at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
   at java.lang.Thread.run(Thread.java:748)

"pool-5-thread-16" #73 prio=5 os_prio=0 tid=0x00007f19f003a800 nid=0x606f waiting on condition [0x00007f1a071f5000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000072bea73b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-5-thread-15" #71 prio=5 os_prio=0 tid=0x00007f1a34008800 nid=0x6068 waiting on condition [0x00007f1a072f6000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000072bea73b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-5-thread-14" #70 prio=5 os_prio=0 tid=0x00007f1a0c02b800 nid=0x6062 waiting on condition [0x00007f1a067eb000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000072bea73b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-5-thread-13" #68 prio=5 os_prio=0 tid=0x00007f1a30390800 nid=0x6059 waiting on condition [0x00007f1a069ed000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000072bea73b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-5-thread-12" #67 prio=5 os_prio=0 tid=0x00007f1a5000c000 nid=0x604f waiting on condition [0x00007f1a068ec000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000072bea73b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-5-thread-11" #66 prio=5 os_prio=0 tid=0x00007f1a38002800 nid=0x6048 waiting on condition [0x00007f19ea777000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000072bea73b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-5-thread-10" #64 prio=5 os_prio=0 tid=0x00007f1a3c01e800 nid=0x6040 waiting on condition [0x00007f1a041e1000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000072bea73b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-5-thread-9" #63 prio=5 os_prio=0 tid=0x00007f1a1800d000 nid=0x603a waiting on condition [0x00007f19ea476000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000072bea73b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-5-thread-8" #62 prio=5 os_prio=0 tid=0x00007f1a34007800 nid=0x6032 waiting on condition [0x00007f1a06cf0000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000072bea73b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-5-thread-7" #55 prio=5 os_prio=0 tid=0x00007f1a306e1800 nid=0x6025 waiting on condition [0x00007f1a06bef000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000072bea73b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"Java2D Disposer" #54 daemon prio=10 os_prio=0 tid=0x00007f19ec08d800 nid=0x6024 in Object.wait() [0x00007f1a06ff3000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
   - locked <0x00000007bb40de98> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
   at sun.java2d.Disposer.run(Disposer.java:148)
   at java.lang.Thread.run(Thread.java:748)

"pool-5-thread-6" #46 prio=5 os_prio=0 tid=0x00007f1a38001800 nid=0x5eae waiting on condition [0x00007f1a04be3000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000072bea73b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"Timer-0" #44 daemon prio=5 os_prio=0 tid=0x00007f19ec060000 nid=0x5cbe in Object.wait() [0x00007f1a070f4000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at java.util.TimerThread.mainLoop(Timer.java:552)
   - locked <0x000000072c1ef4f8> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:505)

"pool-5-thread-5" #43 prio=5 os_prio=0 tid=0x00007f1a18007000 nid=0x5cbd waiting on condition [0x00007f1a06ef2000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000072bea73b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-5-thread-4" #40 prio=5 os_prio=0 tid=0x00007f1a30133800 nid=0x5c0d waiting on condition [0x00007f1a057e5000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000072bea73b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-5-thread-3" #39 prio=5 os_prio=0 tid=0x00007f1a18003800 nid=0x5c06 waiting on condition [0x00007f1a046e2000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000072bea73b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-5-thread-2" #37 prio=5 os_prio=0 tid=0x00007f1a18002000 nid=0x5ad3 waiting on condition [0x00007f1a054e4000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000072bea73b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-2-thread-5" #35 prio=5 os_prio=0 tid=0x00007f1a00009800 nid=0x5acc waiting on condition [0x00007f1a058e6000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x0000000729b6c910> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-2-thread-4" #34 prio=5 os_prio=0 tid=0x00007f1a00008000 nid=0x5acb waiting on condition [0x00007f1a059e7000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x0000000729b6c910> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-5-thread-1" #33 prio=5 os_prio=0 tid=0x00007f1a6566a000 nid=0x5aca waiting on condition [0x00007f1a05ae8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000072bea73b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-2-thread-3" #32 prio=5 os_prio=0 tid=0x00007f1a00006000 nid=0x5ac9 waiting on condition [0x00007f1a05de9000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x0000000729b6c910> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"Blitz-0-Ice.Timer" #31 prio=5 os_prio=0 tid=0x00007f1a64cb0000 nid=0x5ac5 waiting on condition [0x00007f1a066ea000]
   java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000072b69f138> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-2-thread-2" #18 prio=5 os_prio=0 tid=0x00007f1a00003000 nid=0x5ab7 waiting on condition [0x00007f1a075f7000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x0000000729b6c910> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-2-thread-1" #17 prio=5 os_prio=0 tid=0x00007f1a00001800 nid=0x5ab6 waiting on condition [0x00007f1a076f8000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x0000000729b6c910> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"OMERO.scheduler_QuartzSchedulerThread" #15 prio=5 os_prio=0 tid=0x00007f1a6504a000 nid=0x5aa8 waiting on condition [0x00007f1a205fd000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272)

"Ice.ThreadPool.Client-0" #14 prio=5 os_prio=0 tid=0x00007f1a64d80000 nid=0x5aa6 runnable [0x00007f1a4dddc000]
   java.lang.Thread.State: RUNNABLE
   at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
   at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
   at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   - locked <0x0000000726a26358> (a sun.nio.ch.Util$3)
   - locked <0x0000000726a26348> (a java.util.Collections$UnmodifiableSet)
   - locked <0x0000000726a26210> (a sun.nio.ch.EPollSelectorImpl)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   at IceInternal.Selector.select(Selector.java:201)
   at IceInternal.ThreadPool.run(ThreadPool.java:414)
   at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
   at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
   at java.lang.Thread.run(Thread.java:748)

"PostgreSQL-JDBC-SharedTimer-1" #13 daemon prio=5 os_prio=0 tid=0x00007f1a64ca4800 nid=0x5aa5 in Object.wait() [0x00007f1a4e0dd000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at java.lang.Object.wait(Object.java:502)
   at java.util.TimerThread.mainLoop(Timer.java:526)
   - locked <0x00000007260f9950> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:505)

"bitronix-task-scheduler" #12 daemon prio=5 os_prio=0 tid=0x00007f1a64c5f000 nid=0x5aa3 waiting on condition [0x00007f1a4e3f1000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at bitronix.tm.timer.TaskScheduler.run(TaskScheduler.java:245)

"net.sf.ehcache.CacheManager@22d7b4f8" #10 daemon prio=5 os_prio=0 tid=0x00007f1a64bc2800 nid=0x5aa1 in Object.wait() [0x00007f1a4e4f2000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00000007260f9d00> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:552)
   - locked <0x00000007260f9d00> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:505)

"perf4j-async-stats-appender-sink-CoalescingStatistics" #9 daemon prio=5 os_prio=0 tid=0x00007f1a648b5800 nid=0x5a84 waiting on condition [0x00007f1a4ebf3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00000007260654d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
   at org.perf4j.helpers.GenericAsyncCoalescingStatisticsAppender$StopWatchesFromQueueIterator.getNext(GenericAsyncCoalescingStatisticsAppender.java:388)
   at org.perf4j.helpers.GenericAsyncCoalescingStatisticsAppender$StopWatchesFromQueueIterator.hasNext(GenericAsyncCoalescingStatisticsAppender.java:349)
   at org.perf4j.helpers.GroupingStatisticsIterator.getNext(GroupingStatisticsIterator.java:149)
   at org.perf4j.helpers.GroupingStatisticsIterator.hasNext(GroupingStatisticsIterator.java:102)
   at org.perf4j.helpers.GenericAsyncCoalescingStatisticsAppender$Dispatcher.run(GenericAsyncCoalescingStatisticsAppender.java:314)
   at java.lang.Thread.run(Thread.java:748)

"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007f1a640c6800 nid=0x5a01 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f1a640bb800 nid=0x5a00 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f1a640b9000 nid=0x59ff waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f1a640b7800 nid=0x59fe runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f1a64084000 nid=0x59fd in Object.wait() [0x00007f1a548f7000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
   - locked <0x0000000726047c88> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
   at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:212)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f1a6407f000 nid=0x59fc in Object.wait() [0x00007f1a549f8000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at java.lang.Object.wait(Object.java:502)
   at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
   - locked <0x00000007260fb7f0> (a java.lang.ref.Reference$Lock)
   at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=0 tid=0x00007f1a6400c800 nid=0x59f7 in Object.wait() [0x00007f1a6dc99000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x000000072b5bd100> (a IceInternal.ObjectAdapterFactory)
   at java.lang.Object.wait(Object.java:502)
   at IceInternal.ObjectAdapterFactory.waitForShutdown(ObjectAdapterFactory.java:63)
   - locked <0x000000072b5bd100> (a IceInternal.ObjectAdapterFactory)
   at Ice.CommunicatorI.waitForShutdown(CommunicatorI.java:32)
   at ome.services.blitz.Entry.start(Entry.java:202)
   at ome.services.blitz.Entry.main(Entry.java:146)

"VM Thread" os_prio=0 tid=0x00007f1a64077800 nid=0x59fb runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f1a64021800 nid=0x59f8 runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f1a64023000 nid=0x59f9 runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f1a64025000 nid=0x59fa runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007f1a640c9000 nid=0x5a02 waiting on condition

JNI global references: 247



The third output is in the next post, since it is to long for one.
I saw in these output some "lock" and "waiting", maybe this is giving me the trouble.
Thanks again
Alex
alexr
 
Posts: 46
Joined: Tue Jun 12, 2018 12:20 pm

Re: OMERO Bulk import

Postby alexr » Tue Jun 26, 2018 4:23 pm

Here is the missing diagnostics from the last entry:
Code: Select all
Full thread dump OpenJDK 64-Bit Server VM (25.171-b11 mixed mode):

"Attach Listener" #25 daemon prio=9 os_prio=0 tid=0x00007f9a40001000 nid=0x6798 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"pool-2-thread-5" #24 prio=5 os_prio=0 tid=0x00007f9a1800b800 nid=0x6418 waiting on condition [0x00007f9a348af000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000009d0fadb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-2-thread-4" #23 prio=5 os_prio=0 tid=0x00007f9a18009800 nid=0x6417 waiting on condition [0x00007f9a349b0000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000009d0fadb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-2-thread-3" #22 prio=5 os_prio=0 tid=0x00007f9a18007800 nid=0x6416 waiting on condition [0x00007f9a34ab1000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000009d0fadb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"Indexer-0-Ice.Timer" #21 prio=5 os_prio=0 tid=0x00007f9a796b3000 nid=0x6411 waiting on condition [0x00007f9a34624000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
      - parking to wait for  <0x000000009d5a0550> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"Indexer-0-Ice.ThreadPool.Server-0" #20 prio=5 os_prio=0 tid=0x00007f9a79689800 nid=0x6410 runnable [0x00007f9a34bb2000]
   java.lang.Thread.State: RUNNABLE
   at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
   at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
   at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   - locked <0x000000009d5bf540> (a sun.nio.ch.Util$3)
   - locked <0x000000009d5bf530> (a java.util.Collections$UnmodifiableSet)
   - locked <0x000000009d5bec40> (a sun.nio.ch.EPollSelectorImpl)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   at IceInternal.Selector.select(Selector.java:201)
   at IceInternal.ThreadPool.run(ThreadPool.java:414)
   at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
   at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
   at java.lang.Thread.run(Thread.java:748)

"Indexer-0-Ice.ThreadPool.Client-0" #19 prio=5 os_prio=0 tid=0x00007f9a79681000 nid=0x640f runnable [0x00007f9a34cb3000]
   java.lang.Thread.State: RUNNABLE
   at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
   at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
   at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   - locked <0x000000009d59eab0> (a sun.nio.ch.Util$3)
   - locked <0x000000009d59eaa0> (a java.util.Collections$UnmodifiableSet)
   - locked <0x000000009d59e868> (a sun.nio.ch.EPollSelectorImpl)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   at IceInternal.Selector.select(Selector.java:201)
   at IceInternal.ThreadPool.run(ThreadPool.java:414)
   at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
   at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
   at java.lang.Thread.run(Thread.java:748)

"pool-2-thread-2" #18 prio=5 os_prio=0 tid=0x00007f9a18006800 nid=0x640e waiting on condition [0x00007f9a34fb4000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000009d0fadb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"pool-2-thread-1" #17 prio=5 os_prio=0 tid=0x00007f9a18001800 nid=0x640d waiting on condition [0x00007f9a350b5000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000009d0fadb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"OMERO.scheduler_QuartzSchedulerThread" #16 prio=5 os_prio=0 tid=0x00007f9a79528000 nid=0x640c waiting on condition [0x00007f9a353b6000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272)

"metrics-logger-reporter-thread-1" #15 daemon prio=5 os_prio=0 tid=0x00007f9a790cf000 nid=0x640b waiting on condition [0x00007f9a354b7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x000000009e310c08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

"Ice.ThreadPool.Client-0" #14 prio=5 os_prio=0 tid=0x00007f9a78b14000 nid=0x6408 runnable [0x00007f9a36ddd000]
   java.lang.Thread.State: RUNNABLE
   at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
   at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
   at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   - locked <0x000000009a415708> (a sun.nio.ch.Util$3)
   - locked <0x000000009a4156f8> (a java.util.Collections$UnmodifiableSet)
   - locked <0x000000009a4155c0> (a sun.nio.ch.EPollSelectorImpl)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   at IceInternal.Selector.select(Selector.java:201)
   at IceInternal.ThreadPool.run(ThreadPool.java:414)
   at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
   at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
   at java.lang.Thread.run(Thread.java:748)

"net.sf.ehcache.CacheManager@71812481" #12 daemon prio=5 os_prio=0 tid=0x00007f9a78cb3000 nid=0x6406 in Object.wait() [0x00007f9a36ede000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x000000009964f770> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:552)
   - locked <0x000000009964f770> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:505)

"PostgreSQL-JDBC-SharedTimer-1" #11 daemon prio=5 os_prio=0 tid=0x00007f9a785ed800 nid=0x6405 in Object.wait() [0x00007f9a371df000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x0000000099651920> (a java.util.TaskQueue)
   at java.lang.Object.wait(Object.java:502)
   at java.util.TimerThread.mainLoop(Timer.java:526)
   - locked <0x0000000099651920> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:505)

"bitronix-task-scheduler" #10 daemon prio=5 os_prio=0 tid=0x00007f9a78797800 nid=0x6403 waiting on condition [0x00007f9a374e0000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at bitronix.tm.timer.TaskScheduler.run(TaskScheduler.java:245)

"perf4j-async-stats-appender-sink-CoalescingStatistics" #9 daemon prio=5 os_prio=0 tid=0x00007f9a789c7000 nid=0x6401 waiting on condition [0x00007f9a37be1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00000000996186f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
   at org.perf4j.helpers.GenericAsyncCoalescingStatisticsAppender$StopWatchesFromQueueIterator.getNext(GenericAsyncCoalescingStatisticsAppender.java:388)
   at org.perf4j.helpers.GenericAsyncCoalescingStatisticsAppender$StopWatchesFromQueueIterator.hasNext(GenericAsyncCoalescingStatisticsAppender.java:349)
   at org.perf4j.helpers.GroupingStatisticsIterator.getNext(GroupingStatisticsIterator.java:149)
   at org.perf4j.helpers.GroupingStatisticsIterator.hasNext(GroupingStatisticsIterator.java:102)
   at org.perf4j.helpers.GenericAsyncCoalescingStatisticsAppender$Dispatcher.run(GenericAsyncCoalescingStatisticsAppender.java:314)
   at java.lang.Thread.run(Thread.java:748)

"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007f9a78164000 nid=0x63ff runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f9a78161800 nid=0x63fe waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f9a7815e800 nid=0x63fd waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f9a7815d000 nid=0x63fc runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f9a78129800 nid=0x63fb in Object.wait() [0x00007f9a64ea0000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00000000996e0aa8> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
   - locked <0x00000000996e0aa8> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
   at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:212)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f9a78125000 nid=0x63fa in Object.wait() [0x00007f9a64fa1000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00000000996e0d30> (a java.lang.ref.Reference$Lock)
   at java.lang.Object.wait(Object.java:502)
   at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
   - locked <0x00000000996e0d30> (a java.lang.ref.Reference$Lock)
   at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=0 tid=0x00007f9a7800b000 nid=0x63f5 in Object.wait() [0x00007f9a80902000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x000000009d5ba2b8> (a IceInternal.ObjectAdapterFactory)
   at java.lang.Object.wait(Object.java:502)
   at IceInternal.ObjectAdapterFactory.waitForShutdown(ObjectAdapterFactory.java:63)
   - locked <0x000000009d5ba2b8> (a IceInternal.ObjectAdapterFactory)
   at Ice.CommunicatorI.waitForShutdown(CommunicatorI.java:32)
   at ome.services.blitz.Entry.start(Entry.java:202)
   at ome.services.blitz.Entry.main(Entry.java:146)

"VM Thread" os_prio=0 tid=0x00007f9a7811d000 nid=0x63f9 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f9a78020000 nid=0x63f6 runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f9a78021800 nid=0x63f7 runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f9a78023800 nid=0x63f8 runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007f9a78166800 nid=0x6400 waiting on condition

JNI global references: 228


Additionally I checked for locks and get the following:
Code: Select all
gunicorn        21564 FLOCK   0B WRITE 0     0   0 /home/omerouser/omero/tmp/omero_omerouser/21564/.lock
gunicorn        21567 FLOCK   0B WRITE 0     0   0 /home/omerouser/omero/tmp/omero_omerouser/21567/.lock
icebox          23069 POSIX   5B WRITE 0     0   0 /home/omerouser/OMERO.server-5.4.5-ice36-b83/var/master/servers/OMERO.IceStorm/dbs/OMERO.IceStorm/__Freeze/lock
python          23043 FLOCK   0B WRITE 0     0   0 /home/omerouser/omero/tmp/omero_omerouser/23043/.lock
python          23062 FLOCK   0B WRITE 0     0   0 /home/omerouser/omero/tmp/omero_omerouser/23062/.lock
java            23030 POSIX  31B WRITE 0     0   0 /home/omerouser/OMERO.server-5.4.5-ice36-b83/lib/scripts/.omero/repository/9f6b12af-2ef3-4071-9b64-6ab409058ee3/.lock
java            23030 POSIX  31B WRITE 0     0   0 /OMERO/ManagedRepository/.omero/repository/9f6b12af-2ef3-4071-9b64-6ab409058ee3/.lock
gunicorn        21562 FLOCK   0B WRITE 0     0   0 /home/omerouser/omero/tmp/omero_omerouser/21562/.lock
gunicorn        21561 FLOCK   0B WRITE 0     0   0 /home/omerouser/omero/tmp/omero_omerouser/21561/.lock
java            23030 POSIX  31B WRITE 0     0   0 /OMERO/.omero/repository/9f6b12af-2ef3-4071-9b64-6ab409058ee3/.lock
alexr
 
Posts: 46
Joined: Tue Jun 12, 2018 12:20 pm

Re: OMERO Bulk import

Postby jmoore » Tue Jun 26, 2018 8:40 pm

Just some quick feedback this evening: the file locks are as suspected, and the server doesn't appear to be doing anything at all.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: OMERO Bulk import

Postby alexr » Wed Jun 27, 2018 8:09 am

Hi Josh,
I did a complete restart this morning, not only the omero server, but the complete server. Then the locks are gone. When I restart omero and try an other import then I again get only the dataset generated in the server, but no images are added and I end up again with several locks.
I then removed the locks manually and tried the CLI with my above described method using bulk.yml and filelist.tsv. I used the same command that worked yesterday. I get the following output (for one file only):

Code: Select all
==> Summary
0 files uploaded, 0 filesets created, 0 images imported, 0 errors in 0:00:01.369
b failed imports
2018-06-27 09:58:36,581 441        [      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.4.5-ice36-b83
2018-06-27 09:58:36,604 464        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 5.7.3 revision: 34c7ca439a617601ce1ac5dcbdcdcfc6f0bfdaf1 date: 11 January 2018
2018-06-27 09:58:36,680 540        [      main] INFO   formats.importer.cli.CommandLineImporter - Adding exclusion: clientpath
2018-06-27 09:58:36,690 550        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2018-06-27 09:58:36,690 550        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2018-06-27 09:58:36,695 555        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting output format: yaml
2018-06-27 09:58:36,696 556        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2018-06-27 09:58:37,219 1079       [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2018-06-27 09:58:38,771 2631       [      main] WARN   del.enums.handlers.CorrectionEnumHandler - Unknown Correction value 'Achroplan' will be stored as "Other"
2018-06-27 09:58:38,795 2655       [      main] INFO      ome.formats.importer.ImportCandidates - 1 file(s) parsed into 0 group(s) with 1 call(s) to setId in 1565ms. (1576ms total) [0 unknowns]
2018-06-27 09:58:39,134 2994       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to 130.83.98.66:4064
2018-06-27 09:58:39,688 3548       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2018-06-27 09:58:40,129 3989       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.4.5
2018-06-27 09:58:40,129 3989       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.4.5-ice36-b83
2018-06-27 09:58:40,129 3989       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 1.8.0_171
2018-06-27 09:58:40,129 3989       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2018-06-27 09:58:40,129 3989       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2018-06-27 09:58:40,129 3989       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 4.13.0-45-generic
No imports found

Usage:  importer-cli [OPTION]... [path [path ...]]...
   or:   importer-cli [OPTION]... -

Import any number of files into an OMERO instance.
If "-" is the only path, a list of files or directories
is read from standard in. Directories will be searched for
all valid imports.

Session arguments:
  Mandatory arguments for creating a session are 1- either the OMERO server hostname,
username and password or 2- the OMERO server hostname and a valid session key.
  -s SERVER   OMERO server hostname
  -u USER   OMERO username
  -w PASSWORD   OMERO password
  -k KEY   OMERO session key (UUID of an active session)
  -p PORT   OMERO server port (default: 4064)

Naming arguments:
All naming arguments are optional
  -n NAME            Image or plate name to use
  -x DESCRIPTION         Image or plate description to use
  --name NAME            Image or plate name to use
  --description DESCRIPTION      Image or plate description to use

Optional arguments:
  -h               Display this help and exit
  -f               Display the used files and exit
  -c               Continue importing after errors
  -l READER_FILE         Use the list of readers rather than the default
  -d DATASET_ID            OMERO dataset ID to import image into
  -r SCREEN_ID            OMERO screen ID to import plate into
  -T TARGET            target for imports
  --report            Report errors to the OME team
  --upload            Upload broken files and log file (if any) with report. Required --report
  --logs            Upload log file (if any) with report. Required --report
  --email EMAIL            Email for reported errors. Required --report
  --debug LEVEL            Turn debug logging on (optional level)
  --annotation-ns ANNOTATION_NS      Namespace to use for subsequent annotation
  --annotation-text ANNOTATION_TEXT   Content for a text annotation
  --annotation-link ANNOTATION_LINK   Comment annotation ID to link all images to

Examples:

  $ importer-cli -s localhost -u user -w password -d 50 foo.tiff
  $ importer-cli -s localhost -u user -w password -d Dataset:50 foo.tiff
  $ importer-cli -f foo.tiff
  $ importer-cli -s localhost -u username -w password -d 50 --debug ALL foo.tiff

For additional information, see:
https://docs.openmicroscopy.org/latest/omero/users/cli/import.html
Report bugs to <ome-users@lists.openmicroscopy.org.uk>
Import failed with error code: 1. Continuing
c failed imports
2018-06-27 09:58:41,093 395        [      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.4.5-ice36-b83
2018-06-27 09:58:41,114 416        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 5.7.3 revision: 34c7ca439a617601ce1ac5dcbdcdcfc6f0bfdaf1 date: 11 January 2018
2018-06-27 09:58:41,207 509        [      main] INFO   formats.importer.cli.CommandLineImporter - Adding exclusion: clientpath
2018-06-27 09:58:41,217 519        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting checksum algorithm to File-Size-64
2018-06-27 09:58:41,217 519        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
2018-06-27 09:58:41,221 523        [      main] INFO   formats.importer.cli.CommandLineImporter - Setting output format: yaml
2018-06-27 09:58:41,223 525        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2018-06-27 09:58:41,720 1022       [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2018-06-27 09:58:42,161 1463       [      main] WARN   del.enums.handlers.CorrectionEnumHandler - Unknown Correction value 'Achroplan' will be stored as "Other"
2018-06-27 09:58:42,182 1484       [      main] INFO      ome.formats.importer.ImportCandidates - 1 file(s) parsed into 1 group(s) with 1 call(s) to setId in 451ms. (461ms total) [0 unknowns]
2018-06-27 09:58:42,569 1871       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to 130.83.98.66:4064
2018-06-27 09:58:43,174 2476       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2018-06-27 09:58:43,612 2914       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.4.5
2018-06-27 09:58:43,613 2915       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.4.5-ice36-b83
2018-06-27 09:58:43,614 2916       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 1.8.0_171
2018-06-27 09:58:43,614 2916       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2018-06-27 09:58:43,615 2917       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2018-06-27 09:58:43,616 2918       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 4.13.0-45-generic
2018-06-27 09:58:44,174 3476       [      main] INFO          ome.formats.importer.ImportConfig - Using import target: Dataset:name:von Paddy
2018-06-27 09:58:44,293 3595       [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
2018-06-27 09:58:45,076 4378       [      main] ERROR        ome.formats.importer.ImportLibrary - Error on import: could not insert: [ome.model.core.OriginalFile]; SQL [insert into originalfile (atime, ctime, creation_id, external_id, group_id, owner_id, permissions, update_id, hash, hasher, mimetype, mtime, name, path, repo, "size", version, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; constraint [originalfile_repo_path_index]; nested exception is org.hibernate.exception.ConstraintViolationException: could not insert: [ome.model.core.OriginalFile]
omero.ValidationException: null
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_171]
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_171]
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_171]
   at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_171]
   at java.lang.Class.newInstance(Class.java:442) ~[na:1.8.0_171]
   at IceInternal.BasicStream.createUserException(BasicStream.java:2779) ~[ice.jar:na]
   at IceInternal.BasicStream.access$300(BasicStream.java:14) ~[ice.jar:na]
   at IceInternal.BasicStream$EncapsDecoder10.throwException(BasicStream.java:3298) ~[ice.jar:na]
   at IceInternal.BasicStream.throwException(BasicStream.java:2291) ~[ice.jar:na]
   at IceInternal.OutgoingAsync.throwUserException(OutgoingAsync.java:399) ~[ice.jar:na]
   at omero.grid.ManagedRepositoryPrxHelper.end_importFileset(ManagedRepositoryPrxHelper.java:175) ~[blitz.jar:na]
   at omero.grid.ManagedRepositoryPrxHelper.importFileset(ManagedRepositoryPrxHelper.java:46) ~[blitz.jar:na]
   at omero.grid.ManagedRepositoryPrxHelper.importFileset(ManagedRepositoryPrxHelper.java:33) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary.createImport(ImportLibrary.java:374) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:506) ~[blitz.jar:na]
   at ome.formats.importer.ImportLibrary.importCandidates(ImportLibrary.java:289) ~[blitz.jar:na]
   at ome.formats.importer.cli.CommandLineImporter.start(CommandLineImporter.java:289) [blitz.jar:na]
   at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:967) [blitz.jar:na]
2018-06-27 09:58:45,083 4385       [      main] INFO         ome.formats.importer.ImportLibrary - Continuing after error

==> Summary
0 files uploaded, 0 filesets created, 0 images imported, 0 errors in 0:00:01.347
c failed imports


There is the "could not insert: [ome.model.core.OriginalFile]; SQL" error, that I think is the reason I see the datasets generated, but not the actual images.
Thanks
Alex
alexr
 
Posts: 46
Joined: Tue Jun 12, 2018 12:20 pm

PreviousNext

Return to User Discussion

Who is online

Users browsing this forum: No registered users and 1 guest