We're Hiring!

processing time issues

Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats
Please note:
Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats

If you are having trouble with image files, there is information about reporting bugs in the Bio-Formats documentation. Please send us the data and let us know what version of Bio-Formats you are using. For issues with your code, please provide a link to a public repository, ideally GitHub.

processing time issues

Postby cagatay » Sat Jul 29, 2017 6:10 am

I realized that I posted this on the OMERO discussion board instead of bioformats. So moving the question here...

I have a czi file that I am trying to process in batches. Crop all the regions using bftools and then process them using some image processing library. I am using the latest bftools as far as I can tell.

[cagatay@p53 bftools]$ ./bfconvert -version
Version: 5.5.3
Build date: 5 July 2017
VCS revision: 7bed1b04d071667d7b8a2e8c5d42f05fbfc07ecc
[cagatay@p53 bftools]$
------------------------

When I try this on Imagej with bioformats package, after loading the image, I click the crop on load and voila, in a second I get the region in front of me. When I try this on the bfconvert it takes more than a minute per crop operation. I am wondering why the processing times are so different. Given that I'll be running this for the whole image, it will takes days if not weeks to crop all the regions in the image. Any ideas?

[cagatay@p53 bftools]$ export BF_MAX_MEM=50g
[cagatay@p53 bftools]$ ./bfconvert -series 1 -crop 30000,30000,1024,1024 ../x40-01.czi output-512x512-crop.tiff
../x40-01.czi
ZeissCZIReader initializing ../x40-01.czi
[Zeiss CZI] -> output-512x512-crop.tiff [Tagged Image File Format]
Converted 1/1 planes (100%)
[done]
82.68s elapsed (140.0+200.0ms per plane, 82337ms overhead)
[cagatay@p53 bftools]$
cagatay
 
Posts: 3
Joined: Wed Jul 26, 2017 12:39 am

Re: processing time issues

Postby dgault » Mon Jul 31, 2017 1:38 pm

Hi,

I have carried out some profiling of the two scenarios you described and there is indeed a significant difference. From looking at the results it appears that for bfconvert the majority of the time (95%) is actually spent generating and writing out the XML metadata to file, and most of that time is specifically writing the data. When opening in FIJI the plugin is not writing metadata to file until you export it, so that is the main difference between these scenarios.

In saying that it does seem like a very long time to be writing XML data to file. When I load the metadata back in I can again see a very long read time to parse the XML back. I have opened a Trello card on the Bio-Formats inbox for follow investigation into these read and write times: https://trello.com/c/LXSLMzIa/165-long- ... read-times

David Gault
User avatar
dgault
Team Member
 
Posts: 208
Joined: Fri Aug 14, 2015 2:56 pm

Re: processing time issues

Postby cagatay » Tue Aug 01, 2017 6:52 pm

Thank you David,

let me know if I can be of any help. My only other option at this point is to write some fiji plugin to do this in batch mode and I'm trying to avoid that.

Cheers,
Cagatay
cagatay
 
Posts: 3
Joined: Wed Jul 26, 2017 12:39 am

Re: processing time issues

Postby cagatay » Wed Aug 02, 2017 2:37 am

I ended up writing the Fiji plugin if anyone needs it. It's been >10 years since I wrote in Java and this is a quick and dirty solution, so don't judge :)

Code: Select all
import ij.*;
import ij.process.*;
import ij.gui.*;
import java.awt.*;
import ij.plugin.*;
import ij.plugin.frame.*;
import loci.plugins.BF;
import loci.plugins.in.*;
import loci.common.*;

public class My_Plugin implements PlugIn {

   public void run(String arg) {

     String id = "/Users/cbilgin/x40-01.czi";
     try{
      ImporterOptions options = new ImporterOptions();
      options.setId(id);
      options.setAutoscale(true);
      options.setCrop(true);
      options.setCropRegion(0, new Region(60000, 60000, 512, 512));
      options.setColorMode(ImporterOptions.COLOR_MODE_COMPOSITE);
      //...etc.
      ImagePlus[] imps = BF.openImagePlus(options);
      imps[0].show();
     }
     catch(Exception ioex){
      System.out.println(ioex);
      // do nothing!
     }

   }

}
cagatay
 
Posts: 3
Joined: Wed Jul 26, 2017 12:39 am

Re: processing time issues

Postby bramalingam » Wed Aug 02, 2017 4:07 pm

Hi,

Thank you for submitting your code snippet. That looks good to read images.

We have a couple of example code snippets on our bio-formats: ImageJ Overview page,
https://www-legacy.openmicroscopy.org/s ... rs/imagej/

and for your purposes you could take a look at,
https://github.com/openmicroscopy/biofo ... Image.java

This also has an example method for applying lookup tables, similarly you could add a method for cropping images.

For the second part of your workflow (writing the cropped images), please take a look at the following page,
https://www-legacy.openmicroscopy.org/s ... brary.html

This contains example java code snippets for reading and writing images, and specifically :
https://github.com/openmicroscopy/biofo ... ample.java

would give you an idea of how to export cropped regions in an image.

Hope that helps.

Best,
Balaji
User avatar
bramalingam
 
Posts: 70
Joined: Tue Jan 14, 2014 12:01 pm


Return to User Discussion [Legacy]

Who is online

Users browsing this forum: No registered users and 1 guest

cron