We're Hiring!

96 well plate upload - C01 cellomics file format

General user discussion about using the OMERO platform to its fullest. Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

96 well plate upload - C01 cellomics file format

Postby svcostes » Tue May 21, 2013 10:34 pm

Hi everyone,

I have just tested Arrayscan High-Content Analysis microscope from Thermo Scientific (formerly known as cellomics) and I acquired ~2000 images from a 96 well plate. This is ~2GB of images. The images have all the same initial name, and what changes at the end are a few strings identifying the well and the image # for that well. All the images are saved under the same directory. I tried to upload these images via the import GUI of OMERO client. It takes about 1 hour per image to upload... After running it over night, it only loaded 17 images, and created a bunch of orphaned images (thousands) and I cannot access anything. If trying to upload the images from separate directories (one image per directory), then I can upload the image in 6 seconds and it shows up in the screen tab. It seems that OMERO is doing a TOC for each image, as it reads each individual image (my interpretation). I tried command line and same problem...

Except writing a python script to read each image individually, anyone else got this problem and figured a solution?

Thx,

Sylvain
svcostes
 
Posts: 2
Joined: Tue May 21, 2013 10:21 pm

Re: 96 well plate upload - C01 cellomics file format

Postby rleigh » Wed May 22, 2013 8:48 am

Dear Sylvain,

Which directories and/or files did you select to do the import? What happens if you just import one file from the screen? Do the rest then get automatically imported? If the reader is working correctly, it should automatically pick up the entire fileset from just one of the files.

If you are able to zip up and upload the screen privately then we can take a look at the import for you and improve any bugs or performance problems you are seeing. You can do this here: http://qa.openmicroscopy.org.uk/qa/upload/. However, if you have any trouble uploading over http due to the size of the data involved, we can provide you with details for uploading over ssh/sftp.

Kind regards,
Roger
User avatar
rleigh
 
Posts: 217
Joined: Tue Mar 13, 2012 11:45 am

Re: 96 well plate upload - C01 cellomics file format

Postby svcostes » Tue Jun 11, 2013 11:22 pm

Dear Roger,

I tried importing only one file and you are right, part of the problem is that each file upload lets OMERO look into the directory and see all the other file with similar names. Thus, it gets stuck. If you instead load only one file, OMERO also upload all the other files automatically. However, results are very messed up:
1. The original image has a suffix for each channel that looks like: d0.CO1, d1.CO1, d2.CO1 (where d0 is image for channel 0, d1 is image for channel 1, etc...). However the upload ends up creating 3 images with the same name d0.C01. So, once loaded, one cannot differentiate which image is which channel until the image is open and the stain is recognized.
2. All the images ended up being sent to the orphan directory. I then had to copy them into an experiment directory.
3. If I load one image separately by first putting it into a separate directory alone, then OMERO loads the image very quickly and shows where that image is located into a 96 well plate format under the screen tab. So, metadata are completely accurate this way. However, this only works at the single image level.
4. As I tried many times to upload all the images before, the system loaded tens of thousands of images in the orphaned directory. I had to write a script to delete them as OMERO would crash while attempting to do it via the GUI. There are however left overs information I cannot get rid of anymore in OMERO: i.e. screen tabs has a bunch of 96 well plate experiments related to the one experiment I tried to upload many times. These docs are empty and but cannot be deleted... The image tab has also some problems: there are tens of thousand images being listed but I cannot delete them either. While trying to delete, it crashes and I sent the crash log earlier last week.

I am going to start uploading the data for you to look at it.

thx,

Sylvain
svcostes
 
Posts: 2
Joined: Tue May 21, 2013 10:21 pm

Re: 96 well plate upload - C01 cellomics file format

Postby rleigh » Tue Jun 18, 2013 9:14 am

Dear Sylvain,

Thanks for uploading the data. Sorry for the delay in replying--it took a while to test importing and deleting all the data.

I can certainly reproduce the problems you reported here, and I've opened a ticket (#11148) to track this, and you'll be notified of any changes to the ticket. Just a few clarifications needed:

If d0/d1/d2 are the channels, what are o1/o2? Analysis overlays, overview scan, or something else? I think the main problem here is that the CellomicsReader isn't handling these properly.

Regarding the points in your previous post:
All the images ending up in the orphaned images folder are images which we didn't handle importing properly. I agree that the naming here--prefixing with the image selected for import rather than the screen name--is not ideal, but the real problem here is that they are orphaned in the first place! This is something we can look at fixing.

Regarding the deletion of the orphaned images, I'll open a separate ticket about that--the speed is awfully slow and I've also seen odd failures here. I found that moving all the images into a dataset and then deleting the dataset was orders of magnitude faster--I'm just investigating this in more detail.

Regards,
Roger
User avatar
rleigh
 
Posts: 217
Joined: Tue Mar 13, 2012 11:45 am

Re: 96 well plate upload - C01 cellomics file format

Postby rahulmekala » Tue Jun 23, 2015 7:43 am

rleigh wrote:Dear Sylvain,

Which directories and/or files did you select to do the import? What happens if you just import one file from the screen? Do the rest then get automatically imported? If the reader is working correctly, it should automatically pick up the entire fileset from just one of the files.

If you are able to zip up and upload the screen privately then we can take a look at the import for you and improve any bugs or performance problems you are seeing. You can do this here: http://qa.openmicroscopy.org.uk/qa/upload/. However, if you have any trouble uploading over http due to the size of the data involved, we can provide you with details for uploading over ssh/sftp.

Kind regards,
Roger





i am facing with the same problem where i am unable to upload more than 2000 images through import.CLI,it does not work even a single image also, please provide sufficient information to me..
thanks
rahulmekala
 
Posts: 12
Joined: Tue Jun 23, 2015 5:50 am

Re: 96 well plate upload - C01 cellomics file format

Postby mtbc » Tue Jun 23, 2015 9:05 am

Dear Rahul,

Let's get you importing a single image before we work on thousands. :) This thread is about a problem with Cellomics (*.C01 files) and recognizing their channels (original image or analysis results) properly; we fixed it back in 2013. So, I am guessing that your current problem may be something else.

Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Re: 96 well plate upload - C01 cellomics file format

Postby mtbc » Tue Jun 23, 2015 10:25 am

Dear Rahul,

As you mention elsewhere you are using OMERO 5.0: I can confirm that the fix for the problem on this thread was included in Bio-Formats/OMERO 5.0.0.

Cheers,
Mark
User avatar
mtbc
Team Member
 
Posts: 282
Joined: Tue Oct 23, 2012 10:59 am
Location: Dundee, Scotland

Re: 96 well plate upload - C01 cellomics file format

Postby rahulmekala » Wed Jul 15, 2015 10:18 am

Hi all i am getting the error while uploading the cellomics dataset into omero database it is uploading only one image instead of the rest then get automatically imported when selected one image file
If the reader is working correctly, it should automatically pick up the entire fileset from just one of the files.




but it does not happens in my case.
i have decided into three types folder d0,d1,d2 and uploading one folder it uploaded as individual images . each file has the sepate view of having field#1 onlyy.....


please help me
rahulmekala
 
Posts: 12
Joined: Tue Jun 23, 2015 5:50 am

Re: 96 well plate upload - C01 cellomics file format

Postby sbesson » Wed Jul 15, 2015 10:58 am

Hi,

the last post is likely related to the same issue as the one describe in this thread. So let's stick to one thread to minimize confusion.

Reading it, one question occurs to me: can you confirm your Cellomics filesets organized into separate folders with one folder per field? Looking at the CellomicsReader, I think this this data layout is not currently supported as the Cellomics reader expects all files to be part of the same directory.

Best,
Sebastien
User avatar
sbesson
Team Member
 
Posts: 421
Joined: Tue Feb 28, 2012 7:20 pm


Return to User Discussion

Who is online

Users browsing this forum: Google [Bot] and 0 guests