We're Hiring!

InCell 6000 HCS file sets import extremely slowly

Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats
Please note:
Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats

If you are having trouble with image files, there is information about reporting bugs in the Bio-Formats documentation. Please send us the data and let us know what version of Bio-Formats you are using. For issues with your code, please provide a link to a public repository, ideally GitHub.

InCell 6000 HCS file sets import extremely slowly

Postby dsudar » Sun Jun 05, 2016 8:42 pm

Hi all,

At the OME Users meeting I discussed with Melissa my issues with HCS file sets acquired with a GE InCell 6000. Briefly, these are highly multi-location (700 locations per well in 8-well format or 81 locations per well in 96-well format) sets which import into OMERO extremely slowly and consume huge amounts of resources. They do ultimately complete importing so there is no specific error from BioFormats or OMERO itself but was hoping you could assess whether the import performance could be improved. While the import completes, immediately after importing completes (i.e. my CLI import job reports that it's done) the server becomes extremely busy consuming all resources (all 32 cores on the 2 CPUs are at near 100% and all memory appears to be used) so the server becomes unresponsive for any other work. At that point I have to stop the OMERO server and re-start it. After all that, I can see that the data imported correctly and the server is fine.

I have just uploaded to the QA site 2 .zip files. One is a 8-well file set which normally has 700 4-color images per well (in a 20 by 35 grid) but I zipped up only the first location per well. The other is a 96-well file set which normally has 81 2-color images per well (in a 9 by 9 grid) and I also zipped up only the first location per well.

J-marie: these are also 2 examples of the HCS image sets where a "grid per well" viewing option would be important.

Thanks,
- Damir
dsudar
 
Posts: 235
Joined: Mon May 14, 2012 8:43 pm
Location: Berkeley, CA, USA

Re: InCell 6000 HCS file sets import extremely slowly

Postby mlinkert » Mon Jun 06, 2016 10:28 pm

Hi Damir,

Thank you for uploading these example plates.

Based upon initial investigation, I'd expect minimum/maximum pixel value calculation to account for a large portion of the slowness and spike in CPU usage. Could you please confirm the import command used, and in particular whether '--skip minmax' or '--skip all' was used? If not, does using '--skip minmax' help?

The good news is that the minimum and maximum values appear to be pre-calculated and stored in the TIFF files, so we should be able to fix the InCell reader to read these values directly. There is now a ticket:

https://trac.openmicroscopy.org/ome/ticket/13247

and you will be automatically CC'd on any updates.

Regards,
-Melissa
User avatar
mlinkert
Team Member
 
Posts: 353
Joined: Fri May 29, 2009 2:12 pm
Location: Southwest Wisconsin


Return to User Discussion [Legacy]

Who is online

Users browsing this forum: Google [Bot] and 1 guest