Open Microscopy Environment

I'm seeing an issue with DataManagerFacility.saveAndReturnObject failing after the service that has been using this code has been running for a while. (I'm using OMERO 5.3.3 on Linux)

My suspicion is that the session has grown stale, as no traffic has been going to OMERO from the service for a bit, and this causes the above call to fail.

I don't get an Exception, and looking through the DataManagerFacility.saveAndReturnObject code, I think what might be happening is that we hit this part inside handleException:

ConnectionStatus b = getConnectionStatus(t);
if (b != ConnectionStatus.OK)
return;
if (!gateway.isConnected())
return;

In this case the exception is eaten up, and we get back a null pointer from DataManagerFacility.saveAndReturnObject.

Here is how I'm calling the API:

val rv = dm.saveAndReturnObject(ctx, link)

The context (ctx) here has been initialized at service startup with:

Code: Select all

lazy val omeroGateway = new Gateway(new SimpleLogger())

def login(user: String, pass: String, host: String, port: Int): SecurityContext = {
    val cred = new LoginCredentials()
    cred.getServer().setHostname(host)
    cred.getServer().setPort(port)
    cred.getUser().setUsername(user);
    cred.getUser().setPassword(pass);
    val userObj = omeroGateway.connect(cred)
    val ctx = new SecurityContext(userObj.getGroupId())
    ctx
}

lazy val ctx = login(omeroUser, omeroPassword, omeroHost, omeroPort)

Should I try to get a new ctx (login again) and retry, or would there be a better reconnect facility in the omero lib itself that I'm not aware of?

thanks,
thushara

That's interesting. The session actually should be kept alive automatically by the Gateway. But yes, maybe there's a problem, sounds like a reasonable explanation. You're right, unfortunately the exception will be lost in the handleException method in that case, I definitely have to fix that. Do you use a Logger with the Gateway? If so there should be at least a log entry about network issue, 'can't recreate session', etc., although the exception itself is lost.

Internally the Gateway.getConnector(securityContext, true, false) method is called, which tries to recreate the connection in case it got stalled. To pin down the problem a bit, you could call this method yourself before calling DataManagerFacility.saveAndReturnObject and catch the DSOutOfServiceException which will be thrown if the reconnect fails (you don't have to do anything with the returned Connector object, it will be held internally by the Gateway anyway and used in subsequent calls automatically).

Please let me know, if you can find out some more details about this error. Thanks!

Kind Regards,
Dominik

Hi Dominik -
I do use a SimpleLogger with the gateway. But I did not see any log for session closing. I deployed a new version with your suggestion, and will let you know when I see this next.
Best,
thushara

Further update on this:

I was able to reproduce the issue. I think it is an issue with the context (SecurityContext) not being able to communicate if the omero server has since rebooted.

We have a service that on startup establishes a context to the omero server - I think this is no longer valid after omero server has rebooted.

I could try to connect and upon a failure re-login to establish a new context.

You are right, I just tested that, too. A server restart leaves the Gateway in a stale state. Like you said, a workaround for the moment would be to reconnect to the server after a restart.
I already have a PR open to address this (and other) issues, will try to get that into the 5.4.1 release.
https://github.com/openmicroscopy/openm ... /pull/5513

Regards,
Dominik

Hi Dominik -
Did this make into 5.4.1 by any chance?
thx,
thushara

Hi Thushara.
Unfortunately it didn't make it into 5.4.1. But it will be in 5.4.2 which should be released very soon.
Regards,
Dominik