We're Hiring!

cannot upload images after upgrade

Having a problem deploying OMERO? Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

The OMERO.server installation documentation begins here and you can find OMERO.web deployment documentation here.

cannot upload images after upgrade

Postby bernie » Sun Nov 27, 2016 8:53 pm

Hi,

we've just upgrade our omero server from:

omero v 5.1.2 running on centos 6 to omero 5.2.6 running on centos 7

13TB Repository help in attached storage.

The upgrade seemed to go well. Users can logon and view images. OMERO.web works. However users cannot upload new images. Uploads fails almost immediately status 'Preparing upload'.

In Blitz.log I see an associated error:

2016-11-27 20:26:19,697 ERROR [ o.h.e.def.AbstractFlushingEventListener] (.Server-11) Could not synchronize database state with session
org.hibernate.exception.SQLGrammarException: could not insert collection: [ome.model.fs.Fileset.jobLinks#32066]
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:92) ~[hibernate-core.jar:na]
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66) ~[hibernate-core.jar:na]
at org.hibernate.persister.collection.AbstractCollectionPersister.recreate(AbstractCollectionPersister.java:1244) ~[hibernate-core.jar:na]
at org.hibernate.action.CollectionRecreateAction.execute(CollectionRecreateAction.java:58) ~[hibernate-core.jar:na]

e.t.c.

A new test database can be created (using the attached storage as a new repository) with new users and they can import images as normal.

A complication is that the server is in AD in which there is an AD user omero. However, we are using a local omero account using the same uid gid as in the previous omero server before the upgrade, and these ids are different from the AD omero user. The server is configured so that local users take precedence. Also the local omero group as been rename omero_g. I mention this just in case it has a bearing.

We have a short period of time in which we can try and trouble shoot before we should revert to the previous install.

Any help would be greatly appreciated,

Bernie
bernie
 
Posts: 50
Joined: Mon Aug 17, 2009 12:03 pm

Re: cannot upload images after upgrade

Postby jmoore » Mon Nov 28, 2016 6:44 am

Hi Bernie,

bernie wrote:In Blitz.log I see an associated error:

2016-11-27 20:26:19,697 ERROR [ o.h.e.def.AbstractFlushingEventListener] (.Server-11) Could not synchronize database state with session
org.hibernate.exception.SQLGrammarException: could not insert collection:


Can you send this entire file?

A complication is that the server is in AD in which there is an AD user omero. However, we are using a local omero account using the same uid gid as in the previous omero server before the upgrade, and these ids are different from the AD omero user. The server is configured so that local users take precedence. Also the local omero group as been rename omero_g. I mention this just in case it has a bearing.


Is it possible that the upgraded server no longer has write access to the /OMERO (or similar) directories?

Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: cannot upload images after upgrade

Postby bernie » Mon Nov 28, 2016 8:13 am

Good morning Josh,

thanks for your reply. Can you remind me of your upload site please and I'll upload the Blitz.log.

User omero can write to the repository:

bash-4.2$ whoami
omero
bash-4.2$ touch /OMERO/ManagedRepository/test
bash-4.2$ ls -l /OMERO/ManagedRepository/test
-rw-r--r-- 1 omero omero_g 0 Nov 28 08:11 /OMERO/ManagedRepository/test

I also tried a new user but had the same issue,

Bernie
bernie
 
Posts: 50
Joined: Mon Aug 17, 2009 12:03 pm

Re: cannot upload images after upgrade

Postby mtbc » Mon Nov 28, 2016 8:16 am

Dear Bernie,

One can zip the logs up and upload them to http://qa.openmicroscopy.org.uk/qa/upload/

Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Re: cannot upload images after upgrade

Postby bernie » Mon Nov 28, 2016 8:26 am

The server is managed by puppet but this is off.

Also here's the diagnostics:

bash-4.2$ cd ~/OMERO.server
bash-4.2$ bin/omero admin start
No descriptor given. Using etc/grid/default.xml
Waiting on startup. Use CTRL-C to exit
bash-4.2$ bin/omero admin diagnostics

================================================================================
OMERO Diagnostics 5.2.6-ice36-b35
================================================================================

Commands: java -version 1.8.0 (/bin/java)
Commands: python -V 2.7.5 (/bin/python)
Commands: icegridnode --version 3.6.3 (/bin/icegridnode)
Commands: icegridadmin --version 3.6.3 (/bin/icegridadmin)
Commands: psql --version 9.6.1 (/bin/psql)

Server: icegridnode running
Server: Blitz-0 active (pid = 1133, enabled)
Server: DropBox active (pid = 1171, enabled)
Server: FileServer active (pid = 1174, enabled)
Server: Indexer-0 active (pid = 1175, enabled)
Server: MonitorServer active (pid = 1215, enabled)
Server: OMERO.Glacier2 active (pid = 1178, enabled)
Server: OMERO.IceStorm active (pid = 1182, enabled)
Server: PixelData-0 active (pid = 1193, enabled)
Server: Processor-0 active (pid = 1204, enabled)
Server: Tables-0 active (pid = 1213, enabled)
Server: TestDropBox inactive (enabled)

Log dir: /var/omero/OMERO.server-5.2.6-ice36-b35/var/log exists
Log files: Blitz-0.log 140.0 MB errors=450 warnings=112
Log files: DropBox.log 31.0 KB errors=2 warnings=12
Log files: FileServer.log 4.0 KB
Log files: Indexer-0.log 73.0 KB
Log files: MonitorServer.log 55.0 KB
Log files: OMEROweb.lock 0.0 KB
Log files: OMEROweb.log 474.0 KB errors=0 warnings=3027
Log files: OMEROweb_brokenrequest.lock 0.0 KB
Log files: OMEROweb_brokenrequest.log 253.0 KB errors=0 warnings=1512
Log files: PixelData-0.log 54.0 KB
Log files: Processor-0.log 1.0 MB errors=316 warnings=55
Log files: Tables-0.log 26.0 KB errors=4 warnings=8
Log files: TestDropBox.log n/a
Log files: master.err 8.0 KB errors=5 warnings=3
Log files: master.out 0.0 KB
Log files: Total size 142.77 MB


Environment:OMERO_HOME=(unset)
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:OMERO_USERDIR=(unset)
Environment:OMERO_TMPDIR=(unset)
Environment:PATH=/sbin:/bin:/usr/sbin:/usr/bin:/opt/puppetlabs/bin
Environment:PYTHONPATH=(unset)
Environment:ICE_HOME=(unset)
Environment:LD_LIBRARY_PATH=(unset)
Environment:DYLD_LIBRARY_PATH=(unset)

OMERO SSL port:4064
OMERO TCP port:4063
OMERO data dir:'/OMERO' Exists? True Is writable? True
OMERO temp dir:'/var/omero/omero/tmp' Exists? True Is writable? True (Size: 0)

JVM settings: Blitz-${index} -Xmx7200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Indexer-${index} -Xmx4800m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: PixelData-${index} -Xmx7200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Repository-${index} -Xmx4800m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions

OMERO.web status... [ERROR] You are deploying OMERO.web using Apache and mod_wsgi. OMERO.web does not provide any management for the daemon process which communicates with Apache child processes using UNIX sockets to handle a request. Please check Apache directly.
bernie
 
Posts: 50
Joined: Mon Aug 17, 2009 12:03 pm

Re: cannot upload images after upgrade

Postby bernie » Mon Nov 28, 2016 9:54 am

Hi,

I've uploaded the Blitz.log to your upload site.

Interestingly I ran: bin/omero admin cleanse --dry-run /OMERO

I created a new user,' upgrade', to see if a new user would be able to import files (it can't) but the files appear in the list of those that would be removed (as well as some others):

Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/19-05-39.929/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/19-05-41.329/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/19-05-42.364/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/19-23-33.385/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/19-23-34.549/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/19-23-35.610/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/19-24-22.639/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/19-24-23.804/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/19-24-24.847/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/20-24-25.034/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/20-24-26.233/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/20-24-27.361/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/20-26-19.437/
Would remove /OMERO/ManagedRepository/upgrade_2153/2016-11/27/20-40-18.950/

Could it be that omero can read the db but not write to it?

Bernie
bernie
 
Posts: 50
Joined: Mon Aug 17, 2009 12:03 pm

Re: cannot upload images after upgrade

Postby mtbc » Mon Nov 28, 2016 10:11 am

Dear Bernie,

While this isn't of any immediate help to you, I notice that you seem to be running some 9.6 version of PostgreSQL and OMERO 5.2 supports only up to 9.4 as at http://www.openmicroscopy.org/site/supp ... postgresql - I wonder if that's a clue as to your troubles.

Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Re: cannot upload images after upgrade

Postby bernie » Mon Nov 28, 2016 10:21 am

Hi Mark,

Hmm.. I shall drop down to 9.4. I thought I saw that 5.2 supported 9.4+ somewhere which I took to mean 9.4 or above, but I didn't see this, which is unequivocal. Thank you, Bernie
bernie
 
Posts: 50
Joined: Mon Aug 17, 2009 12:03 pm

Re: cannot upload images after upgrade

Postby bernie » Mon Nov 28, 2016 11:33 am

Thanks Mark, dropping the PostgreSQL version down to 9.4.3 fixed the problem,

Bernie
bernie
 
Posts: 50
Joined: Mon Aug 17, 2009 12:03 pm

Re: cannot upload images after upgrade

Postby mtbc » Tue Nov 29, 2016 8:45 am

Dear Bernie,

Thank you very much for letting us know that your OMERO system is now working as it should. Also, my apologies: I was slow to recognize that you probably ran into the issue described at http://www.openmicroscopy.org/site/supp ... s-optional so running the OMERO5.2-5.1-format-syntax-patch.sql script may have more easily solved your problem. While we do not yet do any formal testing with PostgreSQL beyond 9.4 this script provided with OMERO 5.2.6 fixes the only later-PostgreSQL issue we are aware of so far.

Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland


Return to Installation and Deployment

Who is online

Users browsing this forum: No registered users and 1 guest

cron