We're Hiring!

Cleanse issue with 5.2.7

General user discussion about using the OMERO platform to its fullest. Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

Cleanse issue with 5.2.7

Postby jacques2020 » Mon Jun 19, 2017 2:03 pm

Hi,

when calling cleanse in omero 5.2.7 on debian 8 up to date, I get the error reproduced below. It is normally called after backup using the line
Code: Select all
sudo -u cedre-agent $OMERO_HOME/bin/omero -s localhost -u $USERNAME -w "$PASSWORD" -g CeDRE admin cleanse $BINARY_REPOSITORY

cedre-agent is the user running the server and owning omero files. OMERO_HOME is the path (contains a symlink).
I don't see any additional useful indication. Please ask me !

Thank you very much for your help

Best

Jacques


Code: Select all
Created session d6e3cd4c-7c1c-4fc9-aaed-6a6d18e40be5 (root@localhost:4064). Idle timeout: 60 min. Current group: CeDRE
Traceback (most recent call last):
  File "/usr2/local/OMERO/OMERO.server-Current//bin/omero", line 125, in <module>
    rv = omero.cli.argv()
  File "/usr2/local/OMERO/OMERO.server-Current/lib/python/omero/cli.py", line 1438, in argv
    cli.invoke(args[1:])
  File "/usr2/local/OMERO/OMERO.server-Current/lib/python/omero/cli.py", line 952, in invoke
    stop = self.onecmd(line, previous_args)
  File "/usr2/local/OMERO/OMERO.server-Current/lib/python/omero/cli.py", line 1029, in onecmd
    self.execute(line, previous_args)
  File "/usr2/local/OMERO/OMERO.server-Current/lib/python/omero/cli.py", line 1111, in execute
    args.func(args)
  File "/usr2/local/OMERO/OMERO.server-Current/lib/python/omero/cli.py", line 595, in _check_admin
    return func(*args, **kwargs)
  File "/usr2/local/OMERO/OMERO.server-Current/lib/python/omero/plugins/admin.py", line 1796, in cleanse
    dry_run=args.dry_run)
  File "/usr2/local/OMERO/OMERO.server-Current/lib/python/omero/util/cleanse.py", line 250, in cleanse
    delete_empty_dirs(proxy, root, client, dry_run)
  File "/usr2/local/OMERO/OMERO.server-Current/lib/python/omero/util/cleanse.py", line 258, in delete_empty_dirs
    is_empty_dir(repo, '/', False, to_delete)
  File "/usr2/local/OMERO/OMERO.server-Current/lib/python/omero/util/cleanse.py", line 285, in is_empty_dir
    for entry in repo.listFiles(directory):
AttributeError: 'NoneType' object has no attribute 'listFiles'
jacques2020
 
Posts: 102
Joined: Fri Jul 15, 2011 7:46 am

Re: Cleanse issue with 5.2.7

Postby mtbc » Mon Jun 19, 2017 2:40 pm

Dear Jacques,

What output do you get from
Code: Select all
bin/omero fs repos


Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Re: Cleanse issue with 5.2.7

Postby jacques2020 » Mon Jun 19, 2017 3:31 pm

Dear Mark,

here is the output of fs repos, ran as root.

Code: Select all
Created session 8457d830-a219-4502-a43d-4c7a425f6727 (root@localhost:4064). Idle timeout: 60 min. Current group: system
# | Id      | UUID                                 | Type    | Path
---+---------+--------------------------------------+---------+-----------------------------------------------------------
0 | 12      | 2db7d60a-3884-40b6-990d-f89571e95497 | Public  | /OMERO
1 | 13      | ScriptRepo                           | Script  | /usr/local/OMERO/OMERO.server-5.2.7-ice35-b40/lib/scripts
2 | 201     | 47227125-a0dc-4c1c-be62-2ad594f4f6b6 | Public  | /OMERO
3 | 451     | 2e7ed308-e6fa-4fb4-b94a-5e598d993081 | Public  | /OMERO
4 | 850551  | de751af9-105a-48f8-bcff-298462f54c15 | Managed | /OMERO/ManagedRepository
5 | 1171667 | cb2486fd-022d-4e3d-968d-2eb19b67d57e | Public  | /OMERO/ManagedRepository


Thank you for your help !

Best

Jacques
jacques2020
 
Posts: 102
Joined: Fri Jul 15, 2011 7:46 am

Re: Cleanse issue with 5.2.7

Postby jmoore » Tue Jun 20, 2017 9:43 am

Hi Jacques,

it sounds like you are having the same issue as seen in https://www.openmicroscopy.org/community/viewtopic.php?f=5&t=7898&hilit=count+repo#p16220, namely that the OMERO BinaryRepository was moved but the self-correction did not succeed. (Other less likely alternatives are that you ran two OMERO servers against the same directory, etc). Does any of that seem possible?

As a next step, we need to know which repositories are the correct ones. Could you run the following command against your PostgreSQL database?

Code: Select all
SELECT count(id), repo FROM originalfile WHERE mimetype <> 'Repository' AND repo IS NOT NULL GROUP BY repo;


Cheers,
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Cleanse issue with 5.2.7

Postby jacques2020 » Tue Jun 20, 2017 1:29 pm

Hi Josh,

and thank you for your reply.
Here is the output of the command:
Code: Select all
count |                 repo
-------+--------------------------------------
23364 | de751af9-105a-48f8-bcff-298462f54c15
    22 | ScriptRepo
(2 rows)


On a more general perspective,
1) there was only one omero server running on these data at a time but for recovery purpose, I started a second one with a 1 week older snapshot of the database at some point (after stopping the current one). After the recovery, I went back to the current one.
2) The repository was started with version 4 and migrated to version 5.
3) Eventually, I change the hardware containing the data and made a copy at that point. Eventually, the /OMERO path is a symlink pointing to the real mount point of the data NAS. When I upgraded the hardware, the symlink was replaced to point to an other mount point. But it was about 2 years ago while the cleanse issue appeared a few months ago.
I hope it helps to circumvent the possible causes.

Cheers
Jacques
jacques2020
 
Posts: 102
Joined: Fri Jul 15, 2011 7:46 am

Re: Cleanse issue with 5.2.7

Postby jmoore » Tue Jun 20, 2017 2:04 pm

jacques2020 wrote:and thank you for your reply.


Sure. That's what we're here for :)

Here is the output of the command:
Code: Select all
count |                 repo
-------+--------------------------------------
23364 | de751af9-105a-48f8-bcff-298462f54c15
    22 | ScriptRepo
(2 rows)


This is actually quite good. i.e. only one of your repositories have any files associated with it. In a next step, we'll likely be deleting several of the empty repositories.

On a more general perspective,
1) there was only one omero server running on these data at a time but for recovery purpose, I started a second one with a 1 week older snapshot of the database at some point (after stopping the current one). After the recovery, I went back to the current one.
2) The repository was started with version 4 and migrated to version 5.
3) Eventually, I change the hardware containing the data and made a copy at that point. Eventually, the /OMERO path is a symlink pointing to the real mount point of the data NAS. When I upgraded the hardware, the symlink was replaced to point to an other mount point. But it was about 2 years ago while the cleanse issue appeared a few months ago.
I hope it helps to circumvent the possible causes.


Wow! That's quite the list. I'm not sure where to start to figure out which one of the steps may have led to the current situation. But I think the path forward is fairly straight-forward. One more question first:

What is shown by running:

Code: Select all
find /OMERO/.omero -name repo_uuid -exec cat {} \;
find /OMERO/ManagedRepository/.omero -name repo_uuid -exec cat {} \;


Cheers,
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Cleanse issue with 5.2.7

Postby jacques2020 » Tue Jun 20, 2017 2:11 pm

Here are the results of the commands
Code: Select all
find /OMERO/.omero -name repo_uuid -exec cat {} \;
$2e7ed308-e6fa-4fb4-b94a-5e598d993081


and

Code: Select all
find /OMERO/ManagedRepository/.omero -name repo_uuid -exec cat {} \;
$de751af9-105a-48f8-bcff-298462f54c15


Cheers
Jacques
jacques2020
 
Posts: 102
Joined: Fri Jul 15, 2011 7:46 am

Re: Cleanse issue with 5.2.7

Postby jmoore » Tue Jun 20, 2017 2:28 pm

Ok. Good. That points to

Code: Select all
3 | 451     | 2e7ed308-e6fa-4fb4-b94a-5e598d993081 | Public  | /OMERO
4 | 850551  | de751af9-105a-48f8-bcff-298462f54c15 | Managed | /OMERO/ManagedRepository


being your effective (and active) repositories.

Now (as in https://www.openmicroscopy.org/community/viewtopic.php?f=5&t=7898&hilit=count+repo#p16222) it's time to stop your server and take a backup. Once and only once that's done, you'll need to delete the old repositories as in https://www.openmicroscopy.org/community/viewtopic.php?f=5&t=7898&hilit=count+repo&start=10#p16226:

Code: Select all
0 | 12      | 2db7d60a-3884-40b6-990d-f89571e95497 | Public  | /OMERO
2 | 201     | 47227125-a0dc-4c1c-be62-2ad594f4f6b6 | Public  | /OMERO
5 | 1171667 | cb2486fd-022d-4e3d-968d-2eb19b67d57e | Public  | /OMERO/ManagedRepository


which becomes:

Code: Select all
# remove the --dry-run unless you'd like us to look at what will be deleted.
bin/omero delete OriginalFile:12 OriginalFile:201 OriginalFile:1171667 --dry-run --report


~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Cleanse issue with 5.2.7

Postby jacques2020 » Wed Jun 21, 2017 5:58 am

Hello Mark,

The server is used by people in my lab. It is backed up (still running) once a day (config / database / data dir). Do I need a special backup performed without running the server simultaneously ?
Thank you

Cheers

Jacques
jacques2020
 
Posts: 102
Joined: Fri Jul 15, 2011 7:46 am

Re: Cleanse issue with 5.2.7

Postby mtbc » Wed Jun 21, 2017 9:47 am

Dear Jacques,

We would certainly advise backing up during actual downtime in a maintenance window to be sure of taking a cleanly consistent snapshot of the server state. Our documentation ought to make this clearer. Your regular backups will probably suffice but we occasionally recommend going the extra mile because repository deletion / management to recover from unexpected states is exactly the kind of situation that makes data loss rather more likely. We try to be very cautious about helping people to safely preserve their data on their OMERO production systems.

Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Next

Return to User Discussion

Who is online

Users browsing this forum: Google [Bot] and 1 guest

cron