All Imports failing...

[RickB]

Understood, thanks. I'll back up the system per those directions, and not rely purely on VM snapshots...

[RickB]

DB has been dumped, config backed up, and repo+backup files is on NFS mounted NAS... I've taken a snapshot of the VM in addition...

[mtbc]

Rick,

With everything properly backed up and the OMERO server stopped,

Code: Select all

DELETE FROM originalfile AS r WHERE r.id IN (22201, 22251) AND r.mimetype = 'Repository' AND NOT EXISTS (SELECT id FROM originalfile AS f WHERE f.repo = r.hash);



will delete those last two repositories. You may want to try it to see what happens. Honestly, I don't know if it will do good or harm. This has not been extensively tested at our end.

You may instead wish to wait until Monday when others may be able to review this issue. Either way, do have a good weekend, and thank you for your patience.

Cheers,

Mark

[RickB]

Thanks very much, I think I will hold off till Monday before doing this.

[RickB]

I successfully removed the 2 extra repos from the DB, I now only have the 2 repos containing data in the DB:
-bash-4.1$ psql -d omero
psql (9.3.6)

omero=# SELECT count(id), repo FROM originalfile WHERE mimetype <> 'Repository' AND repo IS NOT NULL GROUP BY repo;
count | repo
-------+--------------------------------------
8778 | d1ad86b7-7190-40ea-ac2f-7e4bf68a4edc
16 | ScriptRepo
(2 rows)

However I still see an extra repo via the omero fs repos command, and it looks like a new one that was not present before:

-bash-4.1$ /local/omerouser/OMERO.server/bin/omero fs repos
Previous session expired for root on localhost:4064
Server: [localhost:4064]
Username: [root]

# | Id | UUID | Type | Path
---+-------+--------------------------------------+--------+-----------------------------------------------------------
0 | 14 | ScriptRepo | Script | /local/omerouser/OMERO.server-5.1.2-ice34-b45/lib/scripts
1 | 15 | b69eafe9-ea9e-4b72-8b99-841ae2fdfa39 | Public | /OMERO
2 | 6951 | d1ad86b7-7190-40ea-ac2f-7e4bf68a4edc | Public | /OMERO/ManagedRepository
3 | 23151 | 5197275f-5d47-4ed7-a0e1-115133189771 | Public | /OMERO
(4 rows)


And the end result is the same as before, Omero is up and running, but all imports fail immediately with no visible exception...

[RickB]

One other piece of info - whenever I restart omero via omero admin stop/start, it always complains that a lock file has been left in /OMERO/.omero/repository/e346fa72-0c94-477e-ac75-cdad37293e86/.lock - server starts fine, but I stop the server again, remove lockfile, and restart...

[manics]

Hi Rick

Could you give us some info on your hardware setup, in particular is /OMERO a local disk or network share and if so, what mount options have you used? This might be relevant: https://www.openmicroscopy.org/site/sup ... itory.html

Simon

[RickB]

Thanks Simon, we use an NFS mount for the /OMERO directory/repository - It's an Isilon clustered NAS behind it, and no changes have been done recently to the storage or mount. It's mounted with nfs v3 defaults:

from fstab:
cluster.hpcstg.jax.org:/ifs/data/omero /OMERO nfs defaults 0 0

mount properties:
[root@jaxbhomero01 local]# cat /proc/mounts |grep OMERO
cluster.hpcstg.jax.org:/ifs/data/omero /OMERO nfs rw,relatime,vers=3,rsize=131072,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.9.4.202,mountvers=3,mountport=300,mountproto=udp,local_lock=none,addr=10.9.4.202 0 0

I've looked the permissions over, everything looks good, omerouser:omerouser, 775 throughout the repository directory. Happy to change up the mount options if you think that may be contributing...

[manics]

Do you have a locking daemon running? Could you take a look at these two email list threads:

http://lists.openmicroscopy.org.uk/pipe ... 03502.html
http://lists.openmicroscopy.org.uk/pipe ... 03287.html

Simon

[RickB]

Yes, lockd is running, after reading those pages I moved .omero and ManagedRepository/.omero to /omerolocks and symlinked, no change...