Problem using cleanse

[rdecoster]

When running the cleanse command I get the following error:

Code: Select all

Created session 5bceb0dd-4196-4b41-b15f-3a920ffc357b (root@localhost:4064). Idle timeout: 10 min. Current group: system
Traceback (most recent call last):
File "OMERO.server/bin/omero", line 125, in <module>
rv = omero.cli.argv()
File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1432, in argv
cli.invoke(args[1:])
File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 946, in invoke
stop = self.onecmd(line, previous_args)
File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1023, in onecmd
self.execute(line, previous_args)
File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1105, in execute
args.func(args)
File "/home/omero/OMERO.server/lib/python/omero/plugins/admin.py", line 1764, in cleanse
dry_run=args.dry_run)
File "/home/omero/OMERO.server/lib/python/omero/util/cleanse.py", line 242, in cleanse
delete_empty_dirs(proxy, root, client, dry_run)
File "/home/omero/OMERO.server/lib/python/omero/util/cleanse.py", line 250, in delete_empty_dirs
is_empty_dir(repo, '/', False, to_delete)
File "/home/omero/OMERO.server/lib/python/omero/util/cleanse.py", line 277, in is_empty_dir
for entry in repo.listFiles(directory):
AttributeError: 'NoneType' object has no attribute 'listFiles'

Looking at the blitzlog

Code: Select all

2016-05-18 14:39:21,975 INFO [ ome.services.util.ServiceHandler] (.Server-44) Args: [null, InternalSF@509749142]
2016-05-18 14:39:21,975 INFO [ ome.security.basic.BasicEventContext] (.Server-44) cctx: group=-1
2016-05-18 14:39:21,977 INFO [ ome.security.basic.EventHandler] (.Server-44) Auth: user=0,group=-1,event=null(User),sess=bd8056d6-3ac6-4a76-923f-dae038c3d97a
2016-05-18 14:39:21,987 INFO [ org.perf4j.TimingLogger] (.Server-44) start[1463575161975] time[12] tag[omero.call.success.ome.services.blitz.repo.RepositoryDaoImpl$7.doWork]
2016-05-18 14:39:21,987 INFO [ ome.services.util.ServiceHandler] (.Server-44) Rslt: (ome.model.core.OriginalFile:Id_503)
2016-05-18 14:39:21,993 INFO [ ome.services.util.ServiceHandler] (.Server-41) Executor.doWork -- ome.services.blitz.repo.RepositoryDaoImpl.getOriginalFiles[737c34e2-91e1-4daa-b4b3-b16c5b6400b8, CheckedPath(GBW.GS.ISPAMM.PAVE/u0083649/2016-01/29/14-57-58.145)]
2016-05-18 14:39:21,993 INFO [ ome.services.util.ServiceHandler] (.Server-41) Args: [null, InternalSF@509749142]
2016-05-18 14:39:21,993 INFO [ ome.security.basic.BasicEventContext] (.Server-41) cctx: group=-1
2016-05-18 14:39:21,995 INFO [ ome.security.basic.EventHandler] (.Server-41) Auth: user=0,group=-1,event=null(User),sess=bd8056d6-3ac6-4a76-923f-dae038c3d97a
2016-05-18 14:39:22,006 INFO [ org.perf4j.TimingLogger] (.Server-41) start[1463575161993] time[13] tag[omero.call.success.ome.services.blitz.repo.RepositoryDaoImpl$7.doWork]
2016-05-18 14:39:22,006 INFO [ ome.services.util.ServiceHandler] (.Server-41) Rslt: (ome.model.core.OriginalFile:Id_506)
2016-05-18 14:39:22,012 INFO [ ome.services.util.ServiceHandler] (.Server-42) Executor.doWork -- ome.services.blitz.repo.RepositoryDaoImpl.getOriginalFiles[737c34e2-91e1-4daa-b4b3-b16c5b6400b8, CheckedPath(GBW.GS.ISPAMM.PAVE/u0083649/2016-01/29/14-59-09.678)]
2016-05-18 14:39:22,013 INFO [ ome.services.util.ServiceHandler] (.Server-42) Args: [null, InternalSF@509749142]
2016-05-18 14:39:22,013 INFO [ ome.security.basic.BasicEventContext] (.Server-42) cctx: group=-1
2016-05-18 14:39:22,015 INFO [ ome.security.basic.EventHandler] (.Server-42) Auth: user=0,group=-1,event=null(User),sess=bd8056d6-3ac6-4a76-923f-dae038c3d97a
2016-05-18 14:39:22,026 INFO [ org.perf4j.TimingLogger] (.Server-42) start[1463575162013] time[13] tag[omero.call.success.ome.services.blitz.repo.RepositoryDaoImpl$7.doWork]
2016-05-18 14:39:22,026 INFO [ ome.services.util.ServiceHandler] (.Server-42) Rslt: (ome.model.core.OriginalFile:Id_509)
2016-05-18 14:39:22,031 INFO [ ome.services.util.ServiceHandler] (.Server-43) Executor.doWork -- ome.services.blitz.repo.RepositoryDaoImpl.getOriginalFiles[737c34e2-91e1-4daa-b4b3-b16c5b6400b8, CheckedPath(GBW.GS.ISPAMM.PAVE/u0083649/2016-01/29/15-00-31.758)]
2016-05-18 14:39:22,031 INFO [ ome.services.util.ServiceHandler] (.Server-43) Args: [null, InternalSF@509749142]
2016-05-18 14:39:22,031 INFO [ ome.security.basic.BasicEventContext] (.Server-43) cctx: group=-1
2016-05-18 14:39:22,033 INFO [ ome.security.basic.EventHandler] (.Server-43) Auth: user=0,group=-1,event=null(User),sess=bd8056d6-3ac6-4a76-923f-dae038c3d97a
2016-05-18 14:39:22,043 INFO [ org.perf4j.TimingLogger] (.Server-43) start[1463575162031] time[12] tag[omero.call.success.ome.services.blitz.repo.RepositoryDaoImpl$7.doWork]
2016-05-18 14:39:22,043 INFO [ ome.services.util.ServiceHandler] (.Server-43) Rslt: (ome.model.core.OriginalFile:Id_514)
2016-05-18 14:39:22,049 INFO [o.services.sessions.SessionContext$Count] (.Server-38) -Reference count: bd8056d6-3ac6-4a76-923f-dae038c3d97a=0
2016-05-18 14:39:22,049 INFO [ omero.cmd.SessionI] (.Server-38) cleanupSelf(ServiceFactoryI(session-1d179471-199a-4171-8572-ae68392d9a18/bd8056d6-3ac6-4a76-923f-dae038c3d97a)).
2016-05-18 14:39:22,050 INFO [ omero.cmd.SessionI] (.Server-38) Unregistered servant:bd8056d6-3ac6-4a76-923f-dae038c3d97a/1d179471-199a-4171-8572-ae68392d9a18omero.api.IAdmin(omero.api._IAdminTie@433b4a88)
2016-05-18 14:39:22,050 INFO [ omero.cmd.SessionI] (.Server-38) Removed servant from adapter: 1d179471-199a-4171-8572-ae68392d9a18omero.api.IAdmin
2016-05-18 14:39:22,050 INFO [ omero.cmd.SessionI] (.Server-38) Unregistered servant:bd8056d6-3ac6-4a76-923f-dae038c3d97a/1d179471-199a-4171-8572-ae68392d9a18omero.api.IQuery(omero.api._IQueryTie@25904424)
2016-05-18 14:39:22,050 INFO [ omero.cmd.SessionI] (.Server-38) Removed servant from adapter: 1d179471-199a-4171-8572-ae68392d9a18omero.api.IQuery
2016-05-18 14:39:22,050 INFO [ omero.cmd.SessionI] (.Server-38) Unregistered servant:bd8056d6-3ac6-4a76-923f-dae038c3d97a/1d179471-199a-4171-8572-ae68392d9a18omero.api.IConfig(omero.api._IConfigTie@36940777)
2016-05-18 14:39:22,050 INFO [ omero.cmd.SessionI] (.Server-38) Removed servant from adapter: 1d179471-199a-4171-8572-ae68392d9a18omero.api.IConfig
2016-05-18 14:39:22,050 INFO [ omero.cmd.SessionI] (.Server-38) Unregistered servant:bd8056d6-3ac6-4a76-923f-dae038c3d97a/1d179471-199a-4171-8572-ae68392d9a18omero.grid.SharedResources(omero.grid._SharedResourcesTie@83718b4e)
2016-05-18 14:39:22,050 INFO [ omero.cmd.SessionI] (.Server-38) Removed servant from adapter: 1d179471-199a-4171-8572-ae68392d9a18omero.grid.SharedResources
2016-05-18 14:39:22,050 INFO [ omero.cmd.SessionI] (.Server-38) Unregistered servant:bd8056d6-3ac6-4a76-923f-dae038c3d97a/1d179471-199a-4171-8572-ae68392d9a18omero.api.ISession(omero.api._ISessionTie@4748c27c)
2016-05-18 14:39:22,050 INFO [ omero.cmd.SessionI] (.Server-38) Removed servant from adapter: 1d179471-199a-4171-8572-ae68392d9a18omero.api.ISession
2016-05-18 14:39:25,276 INFO [ ome.services.blitz.fire.SessionManagerI] (.Server-46) Found session locally: 894b7b3b-fdd2-47c5-8bb0-55f62f932868
2016-05-18 14:39:25,277 INFO [ ome.services.blitz.fire.SessionManagerI] (.Server-46) Rejoining session ServiceFactoryI(session-24c03ed2-b9d8-4c33-bb7a-668c671e5bb3/894b7b3b-fdd2-47c5-8bb0-55f62f932868) (agent=OMERO.web)
2016-05-18 14:39:25,282 INFO [o.services.sessions.SessionContext$Count] (.Server-41) -Reference count: 894b7b3b-fdd2-47c5-8bb0-55f62f932868=0
2016-05-18 14:39:25,282 INFO [ omero.cmd.SessionI] (.Server-41) cleanupSelf(ServiceFactoryI(session-24c03ed2-b9d8-4c33-bb7a-668c671e5bb3/894b7b3b-fdd2-47c5-8bb0-55f62f932868)).

I have removed these files in omero but I get the same message with other images. What could be wrong and how can I solve this?
thx in advance.
Best,
Raf

[cblackburn]

Hi Raf,

it sounds as if there may be a problem with the ManagedRepository. can you let me know the output from

Code: Select all

bin/omero fs repos


shows?

I know that this process started with a failed import, have you been available to import an image successfully since that failure?

Cheers,

Colin

[rdecoster]

Code: Select all

# | Id  | UUID                                 | Type    | Path
---+-----+--------------------------------------+---------+------------------------------------------------------
0 | 1   | ScriptRepo                           | Script  | /home/omero/OMERO.server-5.2.1-ice35-b15/lib/scripts
1 | 2   | d578d433-4300-4e76-b8c5-7d60cd5c4a52 | Public  | /media/GBW-0004_CMEVIB_OMERO
2 | 3   | 1e056eb7-7d3e-4a6f-ab97-b44c1233c515 | Public  | /media/GBW-0004_CMEVIB_OMERO/ManagedRepository
3 | 101 | 737c34e2-91e1-4daa-b4b3-b16c5b6400b8 | Managed | /media/GBW-0004_CMEVIB_OMERO/ManagedRepository
4 | 102 | 734d897a-7b3f-4b9a-b81c-720cad61b4ff | Public  | /media/GBW-0004_CMEVIB_OMERO
(5 rows)


We were able to import other images yes. Everything seems normal otherwise.
Best,
Raf

[jmoore]

Hi Raf,

the fact that you have 2 UUIDs registered for the same directory is certainly a surprise. Did you recently move your server? Or is it possible that there are multiple servers running at the same time? (ps auxw | grep java)

Cheers,
~Josh.

[rdecoster]

didn't move the server recently no.

running ps auxw | grep java gives me:

Code: Select all

omero    45852  1.3  6.5 20023364 1078220 ?    Sl   10:16   1:27 java -Xmx12615m -XX:MaxPermSize=4G -XX:+IgnoreUnrecognizedVMOptions -Djava.awt.headless=true -Dlogback.configurationFile=etc/logback.xml -Domero.logfile=var/log/${omero.name}.log -Domero.logbase=/OMERO/ManagedRepository/ -Domero.name=Blitz-0 ome.services.blitz.Entry --Ice.Config=/home/omero/OMERO.server-5.2.1-ice35-b15/var/master/servers/Blitz-0/config/config
omero    45871  1.2  5.1 19961884 847144 ?     Sl   10:16   1:22 java -Xmx12615m -XX:MaxPermSize=4G -XX:+IgnoreUnrecognizedVMOptions -Djava.awt.headless=true -Dlogback.configurationFile=etc/logback-indexing.xml -Domero.logfile=var/log/${omero.name}.log -Domero.name=Indexer-0 -jar lib/server/blitz.jar ome.fulltext --Ice.Config=/home/omero/OMERO.server-5.2.1-ice35-b15/var/master/servers/Indexer-0/config/config
omero    45883 16.6 25.9 20016692 4267744 ?    Sl   10:16  18:45 java -Xmx12615m -XX:MaxPermSize=4G -XX:+IgnoreUnrecognizedVMOptions -Djava.awt.headless=true -Dlogback.configurationFile=etc/logback-indexing.xml -Domero.logfile=var/log/${omero.name}.log -Domero.name=PixelData-0 -jar lib/server/blitz.jar ome.pixeldata --Ice.Config=/home/omero/OMERO.server-5.2.1-ice35-b15/var/master/servers/PixelData-0/config/config


is this ok?
thx,
Raf

[jmoore]

is this ok?

Yup. Looks fine. Could you also try:

Code: Select all

find /OMERO/.omero/ /OMERO/ManagedRepository/.omero -name repo_uuid -exec head {} +


Cheers,
~Josh.

[rdecoster]

Code: Select all

omero@gbw-s-omero01:~$ find /media/GBW-0004_CMEVIB_OMERO/.omero/ /media/GBW-0004_CMEVIB_OMERO/ManagedRepository/.omero -name repo_uuid -exec head {} +
==> /media/GBW-0004_CMEVIB_OMERO/.omero/repository/09791186-dcc1-43c9-915a-7e61e9bfeef6/repo_uuid <==
$734d897a-7b3f-4b9a-b81c-720cad61b4ff
==> /media/GBW-0004_CMEVIB_OMERO/ManagedRepository/.omero/repository/09791186-dcc1-43c9-915a-7e61e9bfeef6/repo_uuid <==
$737c34e2-91e1-4daa-b4b3-b16c5b6400b8

Best,
Raf

[jmoore]

Thanks, Raf. I'm not sure what's going on. At some point in the past, you had "repo_uuid" values in these directories with these uuids:

Code: Select all

1 | 2   | d578d433-4300-4e76-b8c5-7d60cd5c4a52 | Public  | /media/GBW-0004_CMEVIB_OMERO
2 | 3   | 1e056eb7-7d3e-4a6f-ab97-b44c1233c515 | Public  | /media/GBW-0004_CMEVIB_OMERO/ManagedRepository


But those are gone now. Can you also run the query:

Code: Select all

select repo, count(*) from originalfile group by repo;


Cheers,
~Josh.

[rdecoster]

Code: Select all

                 repo                 | count
--------------------------------------+-------
                                      |  2477
737c34e2-91e1-4daa-b4b3-b16c5b6400b8 | 11182
ScriptRepo                           |    17
(3 rows)


does this mean that 2477 files are "repo-less"

thx for your help!
Best,
Raf

[jmoore]

does this mean that 2477 files are "repo-less"

That's ok. You can consider those files "internal" as opposed to those which are uploaded directly from the user into "ManagedRepository".

Judging by your `ps aux` output you restarted today? If so, the problem persists between restarts, right? When did it begin?

~J.