We're Hiring!

bfopen; An unexpected network error occurred

Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats
Please note:
Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats

If you are having trouble with image files, there is information about reporting bugs in the Bio-Formats documentation. Please send us the data and let us know what version of Bio-Formats you are using. For issues with your code, please provide a link to a public repository, ideally GitHub.

bfopen; An unexpected network error occurred

Postby Kouichi_C_Nakamura » Mon Oct 08, 2018 2:09 pm

Although I don't know how to reproduce this, but while I'm using `bfopen.m` of Bio-Formats MATLAB toolbox, I often (but not always) encounter the error below. This is apparently not due to MATLAB code, because the same code works well usually and only after running the code for some time I encounter this. A workaround for this is to restart MATLAB and it always works so far.

But if you could tell what's causing this, and stop this happening, it would be better.

Thanks in advance.

Best,
Kouichi


Code: Select all
Java exception occurred:
java.io.IOException: An unexpected network error occurred
   at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
   at sun.nio.ch.FileDispatcherImpl.pread(Unknown Source)
   at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
   at sun.nio.ch.IOUtil.read(Unknown Source)
   at sun.nio.ch.FileChannelImpl.readInternal(Unknown Source)
   at sun.nio.ch.FileChannelImpl.read(Unknown Source)
   at loci.common.NIOByteBufferProvider.allocateDirect(NIOByteBufferProvider.java:127)
   at loci.common.NIOByteBufferProvider.allocate(NIOByteBufferProvider.java:112)
   at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:629)
   at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:133)
   at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:151)
   at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:165)
   at loci.common.Location.getHandle(Location.java:416)
   at loci.common.Location.getHandle(Location.java:372)
   at loci.common.Location.getHandle(Location.java:353)
   at loci.common.Location.getHandle(Location.java:336)
   at loci.common.Location.checkValidId(Location.java:441)
   at loci.formats.ImageReader.getReader(ImageReader.java:177)
   at loci.formats.ImageReader.setId(ImageReader.java:839)
   at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
   at loci.formats.ChannelFiller.setId(ChannelFiller.java:223)
   at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
   at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:291)

Kouichi_C_Nakamura
 
Posts: 165
Joined: Thu Oct 19, 2017 1:35 pm

Re: bfopen; An unexpected network error occurred

Postby dgault » Tue Oct 09, 2018 9:46 am

If this only occurs after running for a period of time then I suspect that this is caused by a memory issue. It might be worth trying to free up memory from variables after they have been used, using clearvars (https://www.mathworks.com/help/matlab/r ... rvars.html) would be one approach.

Are you running as part of one long continuous program or is multiple runs before the issue occurs?
User avatar
dgault
Team Member
 
Posts: 208
Joined: Fri Aug 14, 2015 2:56 pm

Re: bfopen; An unexpected network error occurred

Postby Kouichi_C_Nakamura » Wed Oct 10, 2018 1:11 pm

Thank you. Because it says network error, I could not associate it with the memory issue. I can have good control of memory in MATLAB workspace using `clear` or `clearvars`. But could it be related to memory usage in Java or anything else that I can't see in MATLAB workspace?

I found an article about "Why does ImageJ not release any memory back to the system?" It seems that the issue is more about Java rather than ImageJ, so I suspect that Bio-Formats can be suffering from the same issue.

http://imagej.net/Troubleshooting#OutOfMemoryError

In that case, `clear` or `clearvars` in MATLAB side wouldn't help. Something like below might?

https://stackoverflow.com/questions/324 ... e-collecti

So far I'm not quite sure, but I think it's more about multiple runs.
Kouichi_C_Nakamura
 
Posts: 165
Joined: Thu Oct 19, 2017 1:35 pm

Re: bfopen; An unexpected network error occurred

Postby dgault » Thu Oct 11, 2018 1:13 pm

It is possible that the issue may also be an error with being able to access and read the file. Is it possible that another program or running code might be locking the file while Bio-Formats is attempting to read it?

Do you have a sample MATLAB program which I could use to try and reproduce the issue? Also are the files you are reading being stored locally on the same machine or are you accessing them remotely?
User avatar
dgault
Team Member
 
Posts: 208
Joined: Fri Aug 14, 2015 2:56 pm

Re: bfopen; An unexpected network error occurred

Postby Kouichi_C_Nakamura » Thu Oct 11, 2018 1:45 pm

File accessibility .... mmm, I can't think of anything particular, though I can't completely deny the possibility. I feel like it doesn't really fit the fact that relaunching MATLAB is always the solution.

I'd love to share the data and code, but at the moment I don't have a good way to reproduce this issue. Once I find one, I'll let you know.
Kouichi_C_Nakamura
 
Posts: 165
Joined: Thu Oct 19, 2017 1:35 pm

Re: bfopen; An unexpected network error occurred

Postby Kouichi_C_Nakamura » Thu Oct 11, 2018 2:12 pm

I've just run the following code to open 3.7 GB TIFF file (14510 * 32510 * 4 pixels, 16 bit), and after around 10 cycles, MATLAB crashed. I could not confirm whether this was the same error as above, but it could.

I had set Java Heap Memory size to 25 GB in advance.

Code: Select all
for i = 1:20

    disp(i)
    b = bfopen(file);

end


If you can get `copytoImagePlus.m` https://github.com/kouichi-c-nakamura/copytoImagePlus and have `ImageJ-MATLAB` working, then you can create the same size of data by the following:

Code: Select all
I = zeros(14510 ,32510,4,'uint16');
imp = copytoImagePlus(I);
ij.IJ.saveAsTiff(imp,file)


I can send you data if you like.
Last edited by Kouichi_C_Nakamura on Mon Oct 15, 2018 9:36 pm, edited 2 times in total.
Kouichi_C_Nakamura
 
Posts: 165
Joined: Thu Oct 19, 2017 1:35 pm

Re: bfopen; An unexpected network error occurred

Postby dgault » Fri Oct 12, 2018 4:04 pm

I can confirm that using the same loop that I have also been able to reproduce the MATLAB crash, though as you mentioned I have not yet been able to verify if this is the same issue as the original error. I will continue testing and debugging this further to see if I can find the root cause of these issues.
User avatar
dgault
Team Member
 
Posts: 208
Joined: Fri Aug 14, 2015 2:56 pm

Re: bfopen; An unexpected network error occurred

Postby Kouichi_C_Nakamura » Fri Oct 12, 2018 5:42 pm

Thanks a lot for the effort.

This may not be an issue of Bio-Formats. Since I've been recently experiencing bizarre behavior of ImageJ Java objects when manipulated from MATLAB. https://uk.mathworks.com/matlabcentral/ ... oning-code

During 35 `for` loop iterations using Bio-Formats and ImageJ from MATLAB, I experienced some 10~15 crashes of MATLAB.
Kouichi_C_Nakamura
 
Posts: 165
Joined: Thu Oct 19, 2017 1:35 pm

Re: bfopen; An unexpected network error occurred

Postby dgault » Tue Oct 16, 2018 3:23 pm

Thanks for the update, this is certainly something we will keep an eye on and continue investigating.
User avatar
dgault
Team Member
 
Posts: 208
Joined: Fri Aug 14, 2015 2:56 pm


Return to User Discussion [Legacy]

Who is online

Users browsing this forum: No registered users and 1 guest

cron