InCell 6000 HCS file sets import extremely slowly

[dsudar]

Hi all,

At the OME Users meeting I discussed with Melissa my issues with HCS file sets acquired with a GE InCell 6000. Briefly, these are highly multi-location (700 locations per well in 8-well format or 81 locations per well in 96-well format) sets which import into OMERO extremely slowly and consume huge amounts of resources. They do ultimately complete importing so there is no specific error from BioFormats or OMERO itself but was hoping you could assess whether the import performance could be improved. While the import completes, immediately after importing completes (i.e. my CLI import job reports that it's done) the server becomes extremely busy consuming all resources (all 32 cores on the 2 CPUs are at near 100% and all memory appears to be used) so the server becomes unresponsive for any other work. At that point I have to stop the OMERO server and re-start it. After all that, I can see that the data imported correctly and the server is fine.

I have just uploaded to the QA site 2 .zip files. One is a 8-well file set which normally has 700 4-color images per well (in a 20 by 35 grid) but I zipped up only the first location per well. The other is a 96-well file set which normally has 81 2-color images per well (in a 9 by 9 grid) and I also zipped up only the first location per well.

J-marie: these are also 2 examples of the HCS image sets where a "grid per well" viewing option would be important.

Thanks,
- Damir

[mlinkert]

Hi Damir,

Thank you for uploading these example plates.

Based upon initial investigation, I'd expect minimum/maximum pixel value calculation to account for a large portion of the slowness and spike in CPU usage. Could you please confirm the import command used, and in particular whether '--skip minmax' or '--skip all' was used? If not, does using '--skip minmax' help?

The good news is that the minimum and maximum values appear to be pre-calculated and stored in the TIFF files, so we should be able to fix the InCell reader to read these values directly. There is now a ticket:

https://trac.openmicroscopy.org/ome/ticket/13247

and you will be automatically CC'd on any updates.

Regards,
-Melissa