We're Hiring!

Issue with Matlab parfor and Bioformats bfGetplane

Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats
Please note:
Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats

If you are having trouble with image files, there is information about reporting bugs in the Bio-Formats documentation. Please send us the data and let us know what version of Bio-Formats you are using. For issues with your code, please provide a link to a public repository, ideally GitHub.

Issue with Matlab parfor and Bioformats bfGetplane

Postby nhuebsch » Sat Jan 28, 2017 7:54 pm

I saw someone else had a problem incorporating Bioformats functions with parfor loops in Matlab. Not sure if this is solveable but I hope so (I also posted on another part of the forum by mistake but I think this is the correct place). I am running a code that processes stacks two images at a time to look at movements. I can use parfor loops if I first load the entire stack of images into ram with bfGetplane. However, if I try to use bfGetplane within a parfor loop, I get this error:

Error using bfGetPlane (line 49)
The value of 'r' is invalid. It must satisfy the function: @(x)isa(x,'loci.formats.IFormatReader')&&~isempty(x.getCurrentFile()).

when the command is : " I = bfGetPlane(reader,i)" (i) = the "ith" plane.

Is there a solution to this issue, or can you recommend a work-around that doesn't involve keeping the entire image series in active RAM?

thanks.
nhuebsch
 
Posts: 7
Joined: Mon Dec 05, 2016 6:31 pm

Re: Issue with Matlab parfor and Bioformats bfGetplane

Postby sbesson » Sun Jan 29, 2017 1:56 pm

Hi,

your error seems to indicate an issue with the initialization of the reader within each parallel loop. Would it be possible for you to share the code or at least the functional block allowing to reproduce your issue?

As Bio-Formats readers are not thread safe, it is not possible to initialize a reader outside a parfor loop and pass it to each parallel worker. Instead, each worker should initialize a Bio-Formats reader, perform work on the image such as reading and processing a particular plane and close the reader. The Bio-Formats MATLAB documentation provides a minimal example of how to use Bio-Formats in a parfor loop.

Let us know if there is anything in this snippet that could be expanded to clarify the use cases

Best,
Sebastien
User avatar
sbesson
Team Member
 
Posts: 421
Joined: Tue Feb 28, 2012 7:20 pm

Re: Issue with Matlab parfor and Bioformats bfGetplane

Postby nhuebsch » Mon Jan 30, 2017 3:42 am

Thank you Sebastian. I will try the instructed method. Here is the code fragment (below). I commented out the way I was doing before (before, I would call up the entire image stack outside parfor, and open individual pieces within).

For this code we are doing block matching optical flow, which is very computationally intensive, hence our need to take advantage of the parallel computing toolbox.

thanks,

Nate

>>>>>
imgPNumber = i;


I1 = stack(:,:,tzerostrain);
imgInit = double(ResizeImage(I1,imageSize, mbSize));

%Test if file is there:
% Can't currently do this test b/c this syntax doesn't work for
% BioFormats
% % if not(exist(reader))
% % error('GetMotionVectors:FileMissing','File missing');
% % end
%I2 = stack(:,:,imgPNumber);
dummy = bfGetPlane(reader,i);
I2 = double(dummy(:,:,1)); % convert from rgb to gray just in case
%
imgNext = double(ResizeImage(I2,imageSize, mbSize));
nhuebsch
 
Posts: 7
Joined: Mon Dec 05, 2016 6:31 pm

Re: Issue with Matlab parfor and Bioformats bfGetplane

Postby sbesson » Mon Jan 30, 2017 7:47 am

Hi Nate,

using the parallel toolbox for block matching makes complete sense. I was able to reproduce your error with the following pseudo-code:

Code: Select all
bfCheckJavaPath();
path = 'test&sizeT=4.fake'; % Test image
% Initialize a reader outside parfor loop
reader = bfGetReader(path);
parfor i =1:4
    % Throws an error
    I= bfGetPlane(reader, i);
    disp(mean(I(:)));
end
reader.close();


As mentioned previously, moving the initialization/closing of the reader steps within the parfor block should be sufficient to run the code in parallel:

Code: Select all
bfCheckJavaPath();
path = 'test&sizeT=4.fake'; % Test image
parfor i =1:4
    % Initialize a reader, retrieve the plane and close the reader
    reader = bfGetReader(path);
    I = bfGetPlane(reader,i);
    disp(mean(I(:)));
    reader.close();
end


Initializing a reader for each parallel execution might become a bottleneck depending on the nature and/or size of the data you are working with. If this turns out to be the case, you might want to cache the initialized reader before running the parallel computation using the Memoizer class as described in the documentation example.

Best,
Sebastien
User avatar
sbesson
Team Member
 
Posts: 421
Joined: Tue Feb 28, 2012 7:20 pm

Re: Issue with Matlab parfor and Bioformats bfGetplane

Postby nhuebsch » Mon Jan 30, 2017 3:12 pm

Thank you Sebastian. Doing the memoization as you suggest works pretty well.

Do you know offhand if the memoization can be done prior to a function call (e.g. so that the function passes the reader object after memoization)? Or must the memoization be done within the same function?

thank you.
nhuebsch
 
Posts: 7
Joined: Mon Dec 05, 2016 6:31 pm

Re: Issue with Matlab parfor and Bioformats bfGetplane

Postby sbesson » Mon Jan 30, 2017 4:00 pm

Hi Nate,

as long as there is no serialization within your function itself (like an internal parfor loop), you should be able to pass the initialized reader to your subfunction as follows:

Code: Select all
bfCheckJavaPath();
path = 'test&sizeT=4.fake'; % Test image
parfor i =1:4
    % Initialize a reader, retrieve the plane and close the reader
    reader = bfGetReader(path);
    runMyAnalysis(reader, i);
    reader.close();
end


Best,
Sebastien
User avatar
sbesson
Team Member
 
Posts: 421
Joined: Tue Feb 28, 2012 7:20 pm


Return to User Discussion [Legacy]

Who is online

Users browsing this forum: No registered users and 0 guests