Omero importer fails to create preview after ugrade to 4.3.0

[jacques2020]

tiffinfo results attached

[cxallan]

Hello,

thanks for the quick reply. However, it's certainly not a problem of the dimension. Rendering and thumbnail generation simply do not work for PNG (and JPEG) files! That is true even for trivial images such as the attached logo, which is of size 260x116, while it works for TIFF files of size 3264x2448. So there seems to be a general problem with importing PNG and JPEG images.
The error message in the INSIGHT log file is the same as the one I posted above. Could it be an issue because the server is version 4.3.0 while the client is version 4.3.2?

Cheers,
Joern

What's actually happening behind the scenes with these images (PNG, JPEG and TIFF) is quite involved, which is why I'm asking the questions. The files themselves are sent to the server where image pyramids are attempted to be generated, if these processes fail you'll have no thumbnails or rendering capabilities. In order to find out what's wrong I'll need your server logs, specifically Blitz-0.log and PixelData-0.log. There's no reason I can see for a problem with the server version difference.

[cxallan]

Hi Chris,

images are 512x512 (I attach output and errors of a tiffinfo).
I also attach the importer.log (the import I mention here started on 10th of August morning).

PixelData has 6 GB RAM (over 24 GB on the server)

/usr/local/OMERO.server-Current/etc/grid$ cat templates.xml | grep -A 10 -B 1 -e "<server-template id=\"PixelDataTemplate\">"

<server-template id="PixelDataTemplate">
<parameter name="index"/>
<parameter name="dir"/>
<parameter name="config" default="default"/>
<server id="PixelData-${index}" exe="${JAVA}" activation="always" pwd="${OMERO_HOME}">
<option>-Xmx6144M</option>
<option>-Djava.awt.headless=true</option>
<option>-Dlog4j.configuration=${OMERO_ETC}log4j-indexing.xml</option>
<option>-Domero.logfile=${OMERO_LOGFILE}</option>
<option>-Domero.name=PixelData-${index}</option>
<option>-jar</option>

Do you have an idea of the issue?

Thanks for our help

Jacques

Thanks Jacques. The importer you're using (4.3.2-DEV) should completely avoid the usage of PixelData-0 for the TIFFs you have. I just took 20110629-Spat-1_1.tif (the test file you provided us) and imported it from my laptop to my laptop which will be similar to your configuration and it took about 5 minutes. The performance numbers you're seeing are horrible. You should be somewhere between 50 and 100 milliseconds per plane, not many-many seconds as you are seeing. Can you send me your Blitz-0.log from the server as well?

[jacques2020]

Hi Chris,

I will of course send Blitz-0.log but it's a more than 400 MB file for yesterday (I have likely a debug option tuned on somewhere?). How can I send it ?

Thanks

Jacques

[toedling]

Hi,
as requested, here are the excerpts from the server's Blitz-0.log and PixelData-0.log that relate to the failed import of the PNG file. Hope these help.
Joern

[cxallan]

Hi Chris,

I will of course send Blitz-0.log but it's a more than 400 MB file for yesterday (I have likely a debug option tuned on somewhere?). How can I send it ?

Thanks

Jacques

You can e-mail it to me if you like, I'll send you a PM with my e-mail address.

Hi,
as requested, here are the excerpts from the server's Blitz-0.log and PixelData-0.log that relate to the failed import of the PNG file. Hope these help.
Joern

Just took a look at these and it looks like your PixelData-0 process has been started with broken configuration. Since it fails to start up correctly it cannot create even the most trivial of pyramids. Could you try bringing your entire OMERO.server cluster of services down, bringing it up cleanly and then having a look at PixelData-0 afterwards to see if it has started up properly? You should see something like this:

Code: Select all

...
2011-08-11 11:33:14,187 INFO  [                ome.services.blitz.Entry] (      main) Creating ome.pixeldata. Please wait...
2011-08-11 11:33:17,230 INFO  [ng.ShutdownSafeEhcacheManagerFactoryBean] (      main) Initializing EHCache CacheManager
2011-08-11 11:33:24,953 INFO  [  ome.services.fulltext.FullTextAnalyzer] (      main) Initialized FullTextAnalyzer
2011-08-11 11:33:41,867 INFO  [                ome.io.nio.PixelsService] (      main) PixelsService(path=/OMERO/, resolver=ome.services.OmeroFilePathResolver@4ca4f040, backoff=ome.io.nio.SimpleBackOff(factor=216.6), sizes=ome.io.nio.ConfiguredTileSizes(w=256,h=256,W=3192,H=3192))
2011-08-11 11:33:41,927 INFO  [  ome.services.pixeldata.PixelDataThread] (      main) Initializing PixelDataThread
2011-08-11 11:33:42,539 INFO  [        ome.services.db.DatabaseIdentity] (      main) Using LSID format: urn:lsid:export.openmicroscopy.org:%s:1e3d1919-9aab-4172-9e80-fcf41aea4e91_%s%s
2011-08-11 11:33:42,835 INFO  [    ome.services.fulltext.FullTextThread] (      main) Initializing Full-Text Indexer
2011-08-11 11:33:42,844 INFO  [    ome.tools.hibernate.ExtendedMetadata] (      main) Calculating ExtendedMetadata...
2011-08-11 11:33:43,185 INFO  [.services.scheduler.SchedulerFactoryBean] (      main) Starting Quartz Scheduler now
2011-08-11 11:33:43,618 INFO  [                ome.services.blitz.Entry] (      main) ome.pixeldata now accepting connections.
...


If that doesn't work to get started I'll need you to give me the output of bin\omero admin diagnostics.

[jacques2020]

Hi Chris,

in the meantime, I discovered that the University offers a file publishing service. You can get the log from
https://servez-vous.univ-rennes1.fr/dzu3sd51bqar

Thank you for your help

Jacques

[toedling]

Hello,
as you suggested that there may be a server problem, I stopped the old server and restarted it again (after upgrading from 4.3.0 to 4.3.1). The pixeldata process seems to have started fine. However, the problem still persists. TIFF files are nicely recognised, the thumbnail is generated within a few seconds and they can be rendered. For PNG files, the thumbnail generation takes very long, for example a few minutes for the small OME logo file that I posted earlier. Once this process is complete, the image can also be rendered. For larger PNG and JPEG pictures, there seems to be no activity, but I guess that within a couple of hours I may or may not have obtain a thumbnail image.
I have attached an archive contain the admin diagnostics output and the log files after restarting the server. Hopefully this can help you to figure out what it wrong with importing JPEG and PNG.
On another note, after upgrading the server from 4.3.0 to 4.3.1 in my Linux Insight client, right-clicking on images seems to have no effect any longer. Is there an obvious explanation for that?
Cheers,
Joern

[cxallan]

Hi Chris,

in the meantime, I discovered that the University offers a file publishing service. You can get the log from
https://servez-vous.univ-rennes1.fr/dzu3sd51bqar

Thank you for your help

Jacques

Thanks Jacques. I think you're actually okay. The most recent information from your log files (the import of 20110629-Spat-1.tif that was started at 2011-08-11 04:24:36) shows the correct timings of approximately 50ms per tile. The rest of the two second or greater tile timings seem to come when your server is under incredible stress and this might just strictly come down to load on your system. This might be made worse by other TIFFs that you imported for which pyramids have been generated.

What the new 4.3.2 importer IS doing is being much more selective about which images have pyramids generated, what it is NOT doing is changing anything that has already been imported. You will have to re-import the TIFFs that have had pyramids generated for them with the 4.3.0 or 4.3.1 importer and DELETE the old images if you do not want those pyramids.

[cxallan]

Hello,
as you suggested that there may be a server problem, I stopped the old server and restarted it again (after upgrading from 4.3.0 to 4.3.1). The pixeldata process seems to have started fine. However, the problem still persists. TIFF files are nicely recognised, the thumbnail is generated within a few seconds and they can be rendered. For PNG files, the thumbnail generation takes very long, for example a few minutes for the small OME logo file that I posted earlier. Once this process is complete, the image can also be rendered. For larger PNG and JPEG pictures, there seems to be no activity, but I guess that within a couple of hours I may or may not have obtain a thumbnail image.
I have attached an archive contain the admin diagnostics output and the log files after restarting the server. Hopefully this can help you to figure out what it wrong with importing JPEG and PNG.
On another note, after upgrading the server from 4.3.0 to 4.3.1 in my Linux Insight client, right-clicking on images seems to have no effect any longer. Is there an obvious explanation for that?
Cheers,
Joern

There's nothing wrong with your JPEG or PNG imports. What you're seeing is a direct result of having had a broken PixelData-0 process for so long. It is having to catch up with all the JPEG or PNG imports you've made to date and TIFF imports that you made with pre-4.3.2 importers across the entire system. It generates pyramids for these images in import order so until that's complete you may indeed have to wait minutes to see a thumbnail.

All that said, we will be extending the size criteria in the 4.3.2 importers so that they also cover JPEG and PNG. This will make imports of these files (if they do not meet the size criteria) similar to the experience you're currently getting with TIFFs. I'll post here as soon as there are builds available that have that functionality.