We're Hiring!

Slow Extracting of Series from Large LIF Files

Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats
Please note:
Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats

If you are having trouble with image files, there is information about reporting bugs in the Bio-Formats documentation. Please send us the data and let us know what version of Bio-Formats you are using. For issues with your code, please provide a link to a public repository, ideally GitHub.

Slow Extracting of Series from Large LIF Files

Postby Cammyron » Tue Apr 11, 2017 10:33 pm

Hi All,

I have some large LIF files (100gb and up) that contain 100-1000 series in them of 100MB each or so. I have a bit of code written for ImageJ that extracts each in turn and saves it. The only issue is it is very very slow. It takes about 2 minutes to open and save each series. The main delay seems to be BioFormats analysing the LIF file each time (i assume building the index of the series in it) before it can open one to save it. Is there a away around this?

Thanks

The code is here
Code: Select all
setBatchMode(true);
fs=File.separator;

pathFile=File.openDialog("Select LIF File to Process");
savePath=getDirectory("Select/Create Output Directory");


run("Bio-Formats Macro Extensions");
Ext.setId(pathFile);
Ext.getSeriesCount(seriesCount);

for(l=43;l<seriesCount;l++){
   Ext.setSeries(l);
   run("Bio-Formats Importer", "open=["+pathFile+"] autoscale color_mode=Default view=Hyperstack stack_order=XYCZT series_"+(l));
   nameStore=getTitle();
   getDateAndTime(year, month, dayOfWeek, dayOfMonth, hour, minute, second, msec);
   print(hour+":"+minute+":"+second+" - Processing Series "+l+" of "+seriesCount);
   //imageName=getInfo("image.filename");
   currentCount=l;
   currentCount=d2s(currentCount,0);
   currentLength=lengthOf(currentCount);
   if(currentLength==1){
      saveName="000"+currentCount;
   }
   if(currentLength==2){
      saveName="00"+currentCount;
   }
   if(currentLength==3){
      saveName="0"+currentCount;
   }
   if(currentLength==4){
      saveName=currentCount;
   }
   
   saveAs("Tiff", savePath+saveName+".tif");
   
   run("Close All");
   
   wait(500);
   call("java.lang.System.gc");
   wait(500);
}
print("DONE!");
Cammyron
 
Posts: 12
Joined: Mon Oct 29, 2012 8:25 pm

Re: Slow Extracting of Series from Large LIF Files

Postby dgault » Wed Apr 12, 2017 2:54 pm

Hi,

One change which may help to improve the performance is to remove the need to run the importer for each series. When using the macro extensions, the call to Ext.setId(pathFile) is enough to initialise the file. Once this is done you can loop through the series count and open the ImagePlus with Ext.openImagePlus.

From the macro you provided this would simply mean replacing:
Code: Select all
run("Bio-Formats Importer", "open=["+pathFile+"] autoscale color_mode=Default view=Hyperstack stack_order=XYCZT series_"+(l));


with:
Code: Select all
Ext.openImagePlus(pathFile);


This should hopefully reduce the long processing times which you are seeing.

David Gault
User avatar
dgault
Team Member
 
Posts: 208
Joined: Fri Aug 14, 2015 2:56 pm

Re: Slow Extracting of Series from Large LIF Files

Postby Cammyron » Wed Apr 12, 2017 11:09 pm

Hi David,

Thanks for the suggestion. Unfortunately hasn't made any difference. It still take 2 min a series to extract and is still analysising the lif file on each loop.

Cheers

Cam
Cammyron
 
Posts: 12
Joined: Mon Oct 29, 2012 8:25 pm

Re: Slow Extracting of Series from Large LIF Files

Postby dgault » Thu Apr 13, 2017 4:19 pm

Hi Cam,

Using macros the other simple approach would be to open a range of series at a time depending on what performance can be handled.

The other alternative would be to create a simple plugin for the operation which would be able to make use of the caching Bio-Formats uses, this would cache the file after the initial initialisation and speed up and future access. Unfortunately this is not currently made use of by the macro extensions but that is something we will look at adding in the future. I have begun putting together a sample of such a plugin which can be found at the link below:
https://github.com/dgault/example-legac ... orter.java

David Gault
User avatar
dgault
Team Member
 
Posts: 208
Joined: Fri Aug 14, 2015 2:56 pm


Return to User Discussion [Legacy]

Who is online

Users browsing this forum: No registered users and 0 guests