We're Hiring!

Replicating an OMERO instance to get TEST environment

Having a problem deploying OMERO? Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

The OMERO.server installation documentation begins here and you can find OMERO.web deployment documentation here.

Replicating an OMERO instance to get TEST environment

Postby rpoehlmann » Wed Jun 08, 2016 8:33 am

Dear all,

I am currently trying to clone/replicate an OMERO instance in order to obtain a TEST environment for rehearsal of OS (CentOS 6->7) and OMERO (5.1.x->5.2.x) version upgrade. While doing so, I stumbled across several issues I would like to address to the community:

  • Is there kind of an official procedure how to clone an instance?
  • How can it be avoided that PROD and cloned TEST instance will automatically build-up an OMERO grid upon service startup? No, I do not want this!
  • How can specific network interfaces be expluded on an OMERO server from being used for internal service connections? E.g. the cloned TEST server has an Infiniband adapter used for connection to a storage device. Apparantly, this one was discovered during OMERO service startup and now also shows up in the node-connection list :?
    Code: Select all
    4901 | BlitzManager -t -e 1.1:tcp -h 10.41.26.240 -p 36663:tcp -h 10.1.26.240 -p 36663:tcp -h 131.152.26.240 -p 36663 |        -1
    20 |                            |       | 2016-06-08 10:07:06.515    | d8396784-53df-4dd4-874c-8f637605c392 |       0 |

    Can this be prevented somehow, e.g. by explicitely excluding the Infiniband interface 10.41.26.240 in the above example from being used?

Many thanks in advance for your help.

Best regards,
-Rainer
User avatar
rpoehlmann
 
Posts: 42
Joined: Thu Feb 09, 2012 2:04 pm

Re: Replicating an OMERO instance to get TEST environment

Postby jmoore » Thu Jun 09, 2016 12:49 pm

rpoehlmann wrote:Dear all,


Hi Rainer,

Is there kind of an official procedure how to clone an instance?


That terminology is not used, but "Backup and Restore is what we use ourselves. This includes migrating the configuration from one installation to the next. Alternatively, we try to use some form of virtualization (docker, virtualbox, openstack) when we know more cloning is necessary.

How can it be avoided that PROD and cloned TEST instance will automatically build-up an OMERO grid upon service startup? No, I do not want this!


OMERO uses a number of ports for finding the various services that are run. If you attempted to start 2 OMERO instances on the same machine, I would expect an error in at least one of them saying that a particular port (like 4063, 4064, or 4061) could not be bound.

How can specific network interfaces be expluded on an OMERO server from being used for internal service connections? E.g. the cloned TEST server has an Infiniband adapter used for connection to a storage device. Apparantly, this one was discovered during OMERO service startup and now also shows up in the node-connection list :?
...
Can this be prevented somehow, e.g. by explicitely excluding the Infiniband interface 10.41.26.240 in the above example from being used?[/list]



The Ice.Default.Host property can be used to pick a single interface. If unset, all interfaces are used.

Code: Select all
bin/omero config set Ice.Default.Host 10.41.26.240


Cheers,
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Replicating an OMERO instance to get TEST environment

Postby rpoehlmann » Thu Jun 09, 2016 4:24 pm

Hi Josh,

thanks a lot for your reply.

That terminology is not used, but "Backup and Restore is what we use ourselves. This includes migrating the configuration from one installation to the next. Alternatively, we try to use some form of virtualization (docker, virtualbox, openstack) when we know more cloning is necessary.


Virtualization is not really "feasible" here since I need to prepare/rehearse for 2 upgrade tasks:
  • migrate the existing productive system onto new hardware
  • upgrade to the most recent OMERO version
Therefore, I wanted to first "backup and restore" the existing setup onto the new hardware, get it running there (as a test, non-productive), then rehearse the OMERO upgrade. If everything is fine, then repeat the procedure but this time real.

OMERO uses a number of ports for finding the various services that are run. If you attempted to start 2 OMERO instances on the same machine, I would expect an error in at least one of them saying that a particular port (like 4063, 4064, or 4061) could not be bound.

Here, we have 2 instances on 2 distinct pieces of hardware. However, when OMERO on the the 2nd hardware (the new server) was started, certain features (image import) did not work correctly and we found from the logs that OMERO no. 2 apparently tries to communicate with OMERO no. 1 although we never ever told it to do so.
Now we are blocking those communication attempts by firewall rules on OMERO no. 1: do not allow any connections from OMERO no. 2. Nevertheless, OMERO no. 2 still tries when importing a new image (which fails!):

Blitz0.log:

2016-06-09 18:08:37,638 INFO [ ome.services.blitz.fire.Registry] (erver-1317) Found 3 repo(s) : [InternalRepository-ScriptRepo -t -e 1.1:tcp -h 131.152.26.240 -p 56624, InternalRepository-7ac6874a-d7b7-4237-888f-8ecef88ddb74 -t -e 1.1:tcp -h 10.1.0.172 -p 52072:tcp -h 131.152.25.172 -p 52072, InternalRepository-f49e4e5d-9854-4ab7-b547-2ca9df884cf6 -t -e 1.1:tcp -h 10.1.0.172 -p 52072:tcp -h 131.152.25.172 -p 52072]

It should find only 1 repo at 131.152.26.240 and nothing else?!? All other IP's (10.1.0.172 + 131.152.25.172)are from OMERO no 1.

OMERO Config (which does not mention any other repo!):
Code: Select all
Ice.Default.Host=131.152.26.240
Ice.IPv6=0
omero.data.dir=/export/omero/OMERO
omero.db.gist=localhost
omero.db.name=omerodb_dev
omero.db.pass=************************************
omero.db.poolsize=80
omero.db.user=omerouser
omero.jvmcfg.max_system_memory=96000
omero.jvmcfg.percent.blitz=30
omero.jvmcfg.percent.indexer=10
omero.jvmcfg.percent.pixeldata=30
omero.jvmcfg.percent.repository=10
omero.jvmcfg.strategy=percent
omero.jvmcfg.system_memory=96000
omero.ldap.base=SOME_LDAP_BASE
omero.ldap.config=true
omero.ldap.password=****************************
omero.ldap.urls=ldap://SOME_LDAP:3268
omero.ldap.user_mapping=omeName=sAMAccountName,firstName=givenName,lastName=sn,email=mail
omero.ldap.username=SOME_NAME
omero.mail.config=true
omero.mail.from=OMERONEW at IMCF <nikolaus.ehrenfeuchter@unibas.ch>
omero.pixeldata.threads=4
omero.sessions.timeout=600000
omero.upgrades.url=
omero.web.admins=[["Rainer Poehlmann", "rainer.poehlmann@unibas.ch"], ["Niko Ehrenfeuchter", "nikolaus.ehrenfeuchter@unibas.ch"]]
omero.web.application_server=fastcgi-tcp
omero.web.apps=["figure", "autotag", "tagsearch"]
omero.web.email_host=smtp.unibas.ch
omero.web.email_subject_prefix=OMERONEW@Biozentrum Notification
omero.web.server_email=omeronew-biozentrum@unibas.ch
omero.web.server_list=[["omeronew.biozentrum.unibas.ch", 4064, "OMERONEW@BZ"]]
omero.web.ui.center_plugins=[["Auto Tag", "autotag/auto_tag_init.js.html", "auto_tag_panel"]]
omero.web.ui.top_links=[["Data", "webindex", {"title": "Browse Data via Projects, Tags etc"}], ["History", "history", {"title": "History"}], ["Help", "http://help.openmicroscopy.org/", {"target": "new", "title": "Open OMERO user guide in a new tab"}], ["Tag Search", "tagsearch"], ["Figure", "figure_index", {"target": "figure", "title": "Open Figure in new tab"}]]
omero.web.webstart_host=omeronew.biozentrum.unibas.ch


Where could this other repo come from? From the database? (I of course dumped/restores the DB from OMERO no. 1 to no. 2). Puzzled!!!

The Ice.Default.Host property can be used to pick a single interface. If unset, all interfaces are used.


Great: I tried it and it worked (see above config list)!

Cheers,
-Rainer
User avatar
rpoehlmann
 
Posts: 42
Joined: Thu Feb 09, 2012 2:04 pm

Re: Replicating an OMERO instance to get TEST environment

Postby jmoore » Thu Jun 09, 2016 4:37 pm

Most importantly:

Great: I tried it and it worked (see above config list)!


Glad to hear it. A theory as to the proxy: when you migrated, did you copy your entire installation, i.e. including the var/ directory?

Cheers,
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Replicating an OMERO instance to get TEST environment

Postby rpoehlmann » Fri Jun 10, 2016 6:13 am

A theory as to the proxy: when you migrated, did you copy your entire installation, i.e. including the var/ directory?

Yes, clearing out only the nginx_tmp and the log directory.

Code: Select all
[omeronas@omero02 var]$ ls -ltr
total 16
lrwxrwxrwx 1 omeronas omeronas   29 Jun 17  2015 log -> /scratch/OMERO.server/var/log
drwxr-x--- 3 omeronas omeronas 4096 Jun 17  2015 registry
drwxr-x--- 5 omeronas omeronas 4096 Jun 17  2015 master
srwxr-xr-x 1 omeronas omeronas    0 May 23 17:35 django_fcgi.sock
-rw-r--r-- 1 omeronas omeronas    6 Jun  8 16:36 django.pid
drwx------ 2 omeronas omeronas 4096 Jun  8 17:44 nginx_tmp


From your question I kind of conclude that this apparently was not a brilliant idea, was it?
Should I give it another try without it, starting over from scratch again?

Thanks,
-Rainer
User avatar
rpoehlmann
 
Posts: 42
Joined: Thu Feb 09, 2012 2:04 pm

Re: Replicating an OMERO instance to get TEST environment

Postby jmoore » Fri Jun 10, 2016 6:38 am

It's an interesting question, Rainer. I've never seen the behavior your suggesting, but on the flip-side, I've never explicitly tested for it. All of var is considered volatile, and in fact there are cases were we suggest removing it. Try deleting it or backing it up (mv var var.bak) as a part of your cloning step. If the problem you were seeing goes away, we'll add an explicit statement to the backup-and-restore page.

Thanks for the heads up!
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Replicating an OMERO instance to get TEST environment

Postby rpoehlmann » Mon Jul 04, 2016 1:12 pm

Hi Josh,
after quite some time of investigation, going back and forth, I guess I now have a clue what was actually happening to cause this issue we observed. And, of course as usual kind of a "self-made" problem! 8-)

Starting scenario as follows:
  • setting up a TEST system to rehearse OS and OMERO version upgrade
  • to also have some data present for the TEST system I rsynced part of the original data repository onto a new volume connected to the TEST server
  • replicating the OMERO installation PROD => TEST by more or less following the official backup & restore procedure
Now, what apparently happened was the following:
  • I forgot to also copy over the repo_uuid from the .omero subdirectory within the ManageRepository folder from PROD to TEST
  • When starting up OMERO, every thing was "fine", images could be displayed (=read), however *no* images could be uploaded
  • Upon the first image upload attempt, OMERO realized that the repository was there but its UUID (a new repo_uuid had been created) did not match the one within the restored database. Error:
    Code: Select all
    ome.formats.importer.ImportLibrary - Error on import: Directory exists but is not registered: CheckedPath(poehlman_2)

  • As a consequence, OMERO TEST started "looking around" for other available ManagedRepositories and found OMERO PROD running in the same subnet!
  • However, upload into this "erroneously clustered" repo on PROD also failed giving a weird error message (unfortunately not the clear one mentioned above I finally found much much later) and misleadingly pointing us into the wrong direction: OMERO spontaneously trying to form clustered services?!?
Everything solved now, I have a TEST system with connected (partially duplicated) TEST repository and even image upload does work!

Sorry again for the long delay in not replying, and thanks again for your help.

Best regards,
-Rainer
User avatar
rpoehlmann
 
Posts: 42
Joined: Thu Feb 09, 2012 2:04 pm


Return to Installation and Deployment

Who is online

Users browsing this forum: No registered users and 1 guest