We're Hiring!

Making sure OMERO python scripts clean up properly

General and open developer discussion about using OMERO APIs from C++, Java, Python, Matlab and more! Please new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

If you are having trouble with custom code, please provide a link to a public repository, ideally GitHub.

Making sure OMERO python scripts clean up properly

Postby graemeball » Thu Jun 23, 2016 4:06 pm

I am writing an OMERO python script for long-running processing and have 2 questions.

Firstly, I am creating a BlitzGateway object:
Code: Select all
conn = BlitzGateway(client_obj=client)

and am wondering if/how I need to close it in a finally block before I do client.closeSession() -- do I need to use .seppuku(), or is this not necessary? (or should I be accessing the gateway in a different way in the first place?)

Secondly, it seems that if I start the script and then log out from my (web) client session, the script dies but my finally block is never reached, and temporary files are therefore not cleaned up. Is this expected behaviour? (or have I made a mistake?) Is there a way I can make sure my cleanup code is always run? If I start the script and close the browser tab without logging out, the session & script continue to run.

Note that I have been testing with a 5.0 server installation, so if there have been updates related to these issues since 5.0, please let me know and I will try with 5.2

Thanks, Graeme
graemeball
 
Posts: 4
Joined: Mon Jan 13, 2014 4:36 pm

Re: Making sure OMERO python scripts clean up properly

Postby atarkowska » Fri Jun 24, 2016 10:18 am

Hi Graeme,

If you call seppuku it will kill omero session which will log user out.
I think your script could use tempfile

Code: Select all
import tempfile
with tempfile.NamedTemporaryFile() as temp:
    temp.write('Some data')
    temp.flush()

this should resolve the issues.

Ola
atarkowska
 
Posts: 327
Joined: Mon May 18, 2009 12:44 pm

Re: Making sure OMERO python scripts clean up properly

Postby graemeball » Fri Jun 24, 2016 11:40 am

Hi Ola,

Thanks for your reply. If tempfile does indeed clean up properly upon user logout (I have not tried), how is the cleanup triggered when logout happens / why does my finally block fail?

I cannot use a simple NamedTemporaryFile, because I need a temporary *directory* in a specific location. I see tempfile module can also create a temporary directory, but:
- other (remote) processes will be writing to this folder (hopefully as the same *user*... definitely same group)
- the python 2 version of 'tempfile' does not appear to clean up temp directories automatically using 'with':
http://stackoverflow.com/questions/6884 ... le-mkdtemp

Cheers,

Graeme
graemeball
 
Posts: 4
Joined: Mon Jan 13, 2014 4:36 pm

Re: Making sure OMERO python scripts clean up properly

Postby atarkowska » Mon Jun 27, 2016 8:59 am

Hi Graeme,

If you want to create shared folder to read from or write to you will have to maintain it separately, for example using crone job. From the other hand did you consider using redis as a persistent store?

Ola
atarkowska
 
Posts: 327
Joined: Mon May 18, 2009 12:44 pm

Re: Making sure OMERO python scripts clean up properly

Postby graemeball » Mon Jun 27, 2016 8:36 pm

OK, so if I understand correctly, OMERO.scripts do not really get a chance to clean up if the client session exits? (is the process they run in killed? I tried digging through the codebase a bit, but I don't think I can grasp it in a reasonable amount of time) It's a pity, but I guess I should be able to clean up the temp directories from the processing node side.

What is more worrying is: if the finally block is never reached, client.closeSession() will never be called. Hopefully though, this is automatically taken care of upon client session logout?

The temp directories will contain both a job description (basically .json), but also image data, which would not fit in a data/object store like redis.

Thanks again for your help,

Graeme
graemeball
 
Posts: 4
Joined: Mon Jan 13, 2014 4:36 pm

Re: Making sure OMERO python scripts clean up properly

Postby atarkowska » Tue Jun 28, 2016 11:05 am

graemeball wrote:OK, so if I understand correctly, OMERO.scripts do not really get a chance to clean up if the client session exits? (is the process they run in killed? I tried digging through the codebase a bit, but I don't think I can grasp it in a reasonable amount of time) It's a pity, but I guess I should be able to clean up the temp directories from the processing node side.

What is more worrying is: if the finally block is never reached, client.closeSession() will never be called. Hopefully though, this is automatically taken care of upon client session logout?


I am not sure if I understand what is your question exactly?
When OMERO.web submit a job to the server to execute script this result in new session creation. Session will be killed and cleaned up regardless. It is true that OMERO.web shouldn't kill running jobs (script) on logout. I already reported that in https://trello.com/c/yc8YNhUa/117-omero ... -on-logout

As you are running 5.0 I am not able to help much, but I tested 5.2 and any exception raised in a body of a script should result in a finally block being called. If you are not sure add more debugging like:

Code: Select all
try:
   ...
except:
    client.setOutput("error", rstring(traceback.format_exc()))
finally:
    client.setOutput("finally",rstring("finally end"))
    client.closeSession()


For more details please look at Processor-0.log

As I said, for those cases where script won't finish sysadmin is responsible for deleting the any files regularly.

graemeball wrote:The temp directories will contain both a job description (basically .json), but also image data, which would not fit in a data/object store like redis.


Definitely JSON can be store in redis either as a plain string in dedicated key (or member/value of a set/list) or in a hash structure.

Ola
atarkowska
 
Posts: 327
Joined: Mon May 18, 2009 12:44 pm


Return to Developer Discussion

Who is online

Users browsing this forum: No registered users and 1 guest