We're Hiring!

Prevent a processing script from timing out

General and open developer discussion about using OMERO APIs from C++, Java, Python, Matlab and more! Please new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

If you are having trouble with custom code, please provide a link to a public repository, ideally GitHub.

Prevent a processing script from timing out

Postby labriedion » Mon Feb 27, 2017 4:39 pm

Hi,

I'm writing a Python script to perform server-side deconvolution. The script downloads images to a temp folder and then these are processed by an imported Python module. The processing could take hours in the case of large Z-stacks. The problem is that I get a timeout error after 1 hour:

Traceback (most recent call last):
File "bin/omero", line 125, in <module>
rv = omero.cli.argv()
File "/home/omero/OMERO.server-5.2.5-ice35-b28/lib/python/omero/cli.py", line 1438, in argv
cli.invoke(args[1:])
File "/home/omero/OMERO.server-5.2.5-ice35-b28/lib/python/omero/cli.py", line 952, in invoke
stop = self.onecmd(line, previous_args)
File "/home/omero/OMERO.server-5.2.5-ice35-b28/lib/python/omero/cli.py", line 1029, in onecmd
self.execute(line, previous_args)
File "/home/omero/OMERO.server-5.2.5-ice35-b28/lib/python/omero/cli.py", line 1111, in execute
args.func(args)
File "/home/omero/OMERO.server-5.2.5-ice35-b28/lib/python/omero/plugins/script.py", line 429, in launch
rv = proc.getResults(3)
File "/home/omero/OMERO.server-5.2.5-ice35-b28/lib/python/omero_Scripts_ice.py", line 929, in getResults
return _M_omero.grid.ScriptProcess._op_getResults.invoke(self, ((waitSecs, ), _ctx))
Ice.UnknownException: exception ::Ice::UnknownException
{
unknown = ome.conditions.SessionTimeoutException: Session (started=2017-02-27 10:25:58.887, hits=296, last access=2017-02-27 11:25:54.553) exceeded timeToLive (3600000) by 61740 ms
at ome.services.sessions.state.SessionCache.getDataNullOrThrowOnTimeout(SessionCache.java:463)
at ome.services.sessions.state.SessionCache.getSessionContext(SessionCache.java:369)
at ome.services.sessions.state.SessionCache.getSessionContext(SessionCache.java:354)
at ome.services.sessions.SessionManagerImpl.getReferenceCount(SessionManagerImpl.java:598)
at ome.services.sessions.SessionManagerImpl.environment(SessionManagerImpl.java:723)
at ome.services.sessions.SessionManagerImpl.outputEnvironment(SessionManagerImpl.java:716)
at omero.grid.InteractiveProcessorI.getResults(InteractiveProcessorI.java:335)
at sun.reflect.GeneratedMethodAccessor1846.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at omero.cmd.CallContext.invoke(CallContext.java:78)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy95.getResults(Unknown Source)
at ome.services.blitz.impl.ScriptProcessI.getResults(ScriptProcessI.java:109)
at sun.reflect.GeneratedMethodAccessor1845.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at omero.cmd.CallContext.invoke(CallContext.java:78)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy117.getResults(Unknown Source)
at omero.grid._ScriptProcessTie.getResults(_ScriptProcessTie.java:118)
at omero.grid._ScriptProcessDisp.___getResults(_ScriptProcessDisp.java:225)
at omero.grid._ScriptProcessDisp.__dispatch(_ScriptProcessDisp.java:324)
at IceInternal.Incoming.invoke(Incoming.java:222)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2482)
at Ice.ConnectionI.dispatch(ConnectionI.java:1258)
at Ice.ConnectionI.message(ConnectionI.java:1213)
at IceInternal.ThreadPool.run(ThreadPool.java:321)
at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:693)
at java.lang.Thread.run(Thread.java:745)


I tried setting a longer timeout interval for scripts (omero.scripts.timeout) but that didn't work. I don't want to change the global session timeout value if I can avoid it.

I also tried setting a timeout loop such as this one: https://github.com/openmicroscopy/openmicroscopy/blob/v.4.4.5/examples/OmeroClients/timeout.py. But that also didn't work, although I'm not sure if it should work or not.

Any ideas?

Thanks,
Étienne
labriedion
 
Posts: 7
Joined: Wed Jan 06, 2016 9:26 pm

Re: Prevent a processing script from timing out

Postby jmoore » Tue Feb 28, 2017 1:57 pm

labriedion wrote:Hi,


Hi Étienne,

I'm writing a Python script to perform server-side deconvolution. The script downloads images to a temp folder and then these are processed by an imported Python module. The processing could take hours in the case of large Z-stacks. The problem is that I get a timeout error after 1 hour:
Code: Select all
...
    unknown = ome.conditions.SessionTimeoutException: Session (started=2017-02-27 10:25:58.887, hits=296, last access=2017-02-27 11:25:54.553) exceeded timeToLive (3600000) by 61740 ms
 


I tried setting a longer timeout interval for scripts (omero.scripts.timeout) but that didn't work. I don't want to change the global session timeout value if I can avoid it.


Several questions: What did you set the timeout to? Did you restart the server after changing the property? What version of the server are you running?

We definitely have examples of the following working:
Code: Select all
omero.scripts.timeout=172800000


Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Prevent a processing script from timing out

Postby labriedion » Tue Feb 28, 2017 3:20 pm

Hi Josh,

Yes, I did change the script timeout value and restarted the server after. I set it at 36000000. I'll try with your value and report back.

My server is 5.2.7 running on CentOS 7.

(Also, an unrelated issue - when I reboot the server and I check the running sessions with "bin/omero admin sessions who", I see lots (30+) of OMERO.scripts sessions starting in a row, is that normal? They go away eventually.)

Thanks,
Étienne
labriedion
 
Posts: 7
Joined: Wed Jan 06, 2016 9:26 pm

Re: Prevent a processing script from timing out

Postby jmoore » Tue Feb 28, 2017 3:30 pm

labriedion wrote:Yes, I did change the script timeout value and restarted the server after. I set it at 36000000. I'll try with your value and report back.


Thanks.

My server is 5.2.7 running on CentOS 7.


Noted.

(Also, an unrelated issue - when I reboot the server and I check the running sessions with "bin/omero admin sessions who", I see lots (30+) of OMERO.scripts sessions starting in a row, is that normal? They go away eventually.)


Yes, this is normal. That's the server calculating the params for each of the scripts by calling out to Python.

Cheers,
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Prevent a processing script from timing out

Postby labriedion » Tue Feb 28, 2017 5:45 pm

Hi Josh,

It actually worked with your value! I guess I must've set the value wrong the first time or it wasn't long enough.

Thanks a lot!

Étienne
labriedion
 
Posts: 7
Joined: Wed Jan 06, 2016 9:26 pm

Re: Prevent a processing script from timing out

Postby jmoore » Tue Feb 28, 2017 5:55 pm

Glad to hear it!
~J.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany


Return to Developer Discussion

Who is online

Users browsing this forum: No registered users and 0 guests