ClayB wrote:jmoore wrote:A couple of questions before a full response:
- From where to where are you moving the data.dir?
The move is from the original location specified during installation (/mnt/app_hdd/omero/omero_server) to /cluster_share/tools/imaging/OMERO.
Makes sense. And there's to change to ${omero.data.dir} itself, correct?
- What type of file systems are involved?
The original site is EXT4 while the target file system is Lustre.
Thanks. I was worried that we were running into NFS issues.
- Can you describe the exact steps you took?
a. Create OMERO directory in /cluster_share
> mkdir /cluster_share/tools/imaging/OMERO
b. Copy current ManagedRepository to shared area
> cp -r omero_server/ManagedRepository /cluster_share/tools/imaging/OMERO
c. Configure OMERO server to point at new MR location
> OMERO.server/bin/omero config set omero.managed.dir /cluster_share/tools/imaging/OMERO/ManagedRepository
d. Restart OMERO server (with new location of MR)
> OMERO.server/bin/omero admin restart
Thanks for the detailed steps, Clay! I've tried to reproduce with the following:
- Code: Select all
# default.sh
NAME=ome9
OMERO=`pwd`/dist/bin/omero
rm -rf `pwd`/dist/var
$OMERO admin stop
set -e
set -u
$OMERO version
dropdb $NAME
createdb $NAME
$OMERO db script --password ome -f- | psql $NAME
rm -rf /tmp/$NAME
mkdir /tmp/$NAME
cd /tmp/$NAME
mkdir data
$OMERO config set omero.data.dir `pwd`/data
$OMERO admin start
$OMERO admin waitup
$OMERO -s root@localhost -w ome fs repos
and
- Code: Select all
# copied.sh
set -e
set -u
NAME=ome9
OMERO=`pwd`/dist/bin/omero
$OMERO admin stop
cd /tmp/$NAME
COPIED=`pwd`/copied.dir/OMERO
mkdir -p $COPIED
$OMERO config set omero.managed.dir $COPIED/ManagedRepository
cp -r data/ManagedRepository $COPIED
$OMERO admin start
$OMERO admin waitup
$OMERO -s root@localhost -w ome fs repos
But on doing so, I see this in my logs:
- Code: Select all
/opt/ome9$ grep "updated to" dist/var/log/Blitz-0.log
2015-06-30 11:30:11,841 WARN [ o.s.blitz.repo.AbstractRepositoryI] (2-thread-3) Data directory moved: /tmp/ome9/data/ManagedRepository updated to /tmp/ome9/copied.dir/OMERO/ManagedRepository
There may be similar issues with Lustre. Could you attach your logs zipped? (I'm wondering if there are any other WARNs or ERRORs)
It might also be useful to have a jstack output from the Blitz process:
- Code: Select all
jstack $(bin/omero admin ice server pid Blitz-0)
If this is related to the filesystem & locking, then likely you will need to move /cluster_share/tools/imaging/OMERO/ManagedRepository/.omero onto a non-Lustre file system unless there's someone who can fix locking directly in Lustre itself.
ClayB wrote:The move is necessary since the compute nodes in the cluster don't have access to /mnt/app_hdd, but do have access to everything in the /cluster_share system.
That also makes sense. If this isn't a file locking issue as it is with NFS, then perhaps you could either:
- create the ManagedRepository directory yourself and set the property before your first startup?
- use a symlink from the old location to the new? (omero_server/ManagedRepository -> /cluster/ ....) and not set the propery?
Thanks for helping us to track this down.
~Josh.