OMERO and data from Perkin Elmer Opera Phenix

[mtbc]

Dear Frank,

Thank you for the information. That's a puzzle. Would you be able to zip up your OMERO server logs from its var/log/ directory and submit them to https://www.openmicroscopy.org/qa2/qa/upload/? Perhaps there is a clue in that as to what is going on after file upload.

Cheers,
Mark

[frankgu968]

Hi Mark,

I have uploaded the logs for your reference.

I have also run the

Code: Select all

omero fs images

command and the output is as follows:

Code: Select all

# | Image | Name | FS | # Files | Size
---+-------+------+----+---------+------
(0 rows, starting at 0 of approx. 0)

Checking the output from the import process, I see this set of interesting DEBUG-level messages:

Code: Select all

2018-12-11 22:53:54,945 601        [      main] DEBUG     ome.formats.importer.cli.ErrorHandler - SCANNING: Depth:2 Num: 11000 Tot:  n/a File: h2sk1fk1fl1.tiff
2018-12-11 22:53:54,946 602        [      main] DEBUG     ome.formats.importer.cli.ErrorHandler - SCANNING: Depth:2 Num: 11100 Tot:  n/a File: h1sk1fk1fl1.tiff
2018-12-11 22:53:54,946 602        [      main] DEBUG     ome.formats.importer.cli.ErrorHandler - SCANNING: Depth:2 Num: 11200 Tot:  n/a File: h3sk1fk1fl1.tiff
2018-12-11 22:53:54,946 602        [      main] DEBUG     ome.formats.importer.cli.ErrorHandler - SCANNING: Depth:2 Num: 11300 Tot:  n/a File: h2sk1fk1fl1.tiff
[A LOT OF SIMILAR ENTRIES LATER]
2018-12-12 00:07:22,104 4407760    [      main] TRACE                      loci.common.Location - Location is not a URL
2018-12-12 00:07:22,104 4407760    [      main] TRACE                      loci.common.Location - getParent()
2018-12-12 00:07:22,104 4407760    [      main] TRACE                      loci.common.Location - Location(/mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11/Images)
2018-12-12 00:07:22,104 4407760    [      main] TRACE                      loci.common.Location - Location is not a URL
2018-12-12 00:07:22,104 4407760    [      main] TRACE                      loci.common.Location - getAbsolutePath()
2018-12-12 00:07:22,104 4407760    [      main] TRACE                      loci.common.Location - Location(/mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11/Images/r04c09f08p73-ch1sk1fk1fl1.tiff)
2018-12-12 00:07:22,104 4407760    [      main] TRACE                      loci.common.Location - Location is not a URL
2018-12-12 00:07:22,104 4407760    [      main] TRACE                      loci.common.Location - getAbsolutePath()
2018-12-12 00:07:22,104 4407760    [      main] TRACE                      loci.common.Location - Location(/mnt/disk/OMERO/raw_data/OncogenePL3_B8-G11/Images/Index.idx.xml)

Please note that for this upload, our server ran out of disk space due to storing the huge debug log from the upload, so this may be reflected in the logs.

Meanwhile, I'll continue debugging with the logs. Thanks for looking into this for us!

Regards,
Frank

[mtbc]

Dear Frank,

There are indeed interesting log entries, thank you. master.err has an Ice::ConnectionRefusedException followed by a Ice.CommunicatorDestroyedException for a projection query occurring in verifyUpload which checks the file checksums but it looks as if the server was shut down at that point on Tuesday afternoon. Was this done because of the stalled import or did the server perhaps restart by itself? The verification should have gone quickly because your "--skip all" effects "--checksum_algorithm File-Size-64". It could also be interesting to keep an eye on your PostgreSQL server's state during this in case the holdup is somehow in that transaction.

Cheers,
Mark

[mtbc]

Dear Frank,

Just thinking, with your use of the --parallel options: how do you have omero.db.poolsize set?

Cheers,
Mark

[frankgu968]

Hi Mark,

Thanks for analyzing the log files. I've taken a look at the points you mentioned in the log. That entry must have been from a previous upload on Dec 11th that I have manually stopped, and then rebooted the server for good measure. I then attempted to do another import directly after the reboot. It's puzzling why absolutely nothing appears on OMERO; not even "orphaned images"...

I've attached my omero configs for your reference

Code: Select all

omero.data.dir=/mnt/disk/OMERO
omero.db.name=*
omero.db.pass=*
omero.db.poolsize=100
omero.db.user=*
omero.jvmcfg.heap_size=20G
omero.jvmcfg.max_system_memory=20480
omero.jvmcfg.percent.blitz=90
omero.jvmcfg.perm_gen=8G
omero.jvmcfg.strategy=percent
omero.sessions.timeout=3600000
omero.threads.background_threads=16

I've just tried to maximize the amount of resources given to OMERO.server. If you have any suggestions with regards to the above, that would be very helpful too!

Thanks,
Frank

[mtbc]

Dear Frank,

Thank you, that is certainly helpful. Does it help if you reduce the scale of your parallel CLI import options? Specifically I wonder if it is important to reduce --parallel-fileset comfortably below your omero.threads.background_threads setting but if that doesn't help then maybe try much reducing --parallel-upload too. That may be something for which our documentation needs more work.

Cheers,
Mark

[frankgu968]

Hi Mark,

I have tried that as well. I have tried the import even without parallel settings. I have also tried to tweak the number of background threads settings to no avail... On that note, I found that setting the parallel upload option to about 2x the number of background_threads yields the lowest import time; for my machine at least.

During import, the CPU and RAM are at a comfortable utilization level (50%); there is a low amount of disk activity as well (I'm using the ln_s option).

Is there anything else we can try at this point?

Regards,
Frank

[Dominik]

Hi Frank,

as many people are already away in their Christmas holidays, you probably won't get a reply this week. But we will come back to this issue as soon as possible after the holidays!

Kind Regards,
Dominik

[frankgu968]

Hi Dominik,

Thanks for the heads up. Hope you all have a great holiday!

Thank,
Frank

[jmoore]

Hi Frank,

Unfortunately, we haven't had any new insights with our holiday-recharged batters. Would it be possible for you to upload one of the datasets so that we can reproduce the issue you are seeing? If so, I'll send along FTP instructions (assuming it's bigger than 2GB).

All the best,
~Josh