We're Hiring!

Opening files while they are being written

Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats
Please note:
Historical discussions about the Bio-Formats library. Please look for and ask new questions at https://forum.image.sc/tags/bio-formats

If you are having trouble with image files, there is information about reporting bugs in the Bio-Formats documentation. Please send us the data and let us know what version of Bio-Formats you are using. For issues with your code, please provide a link to a public repository, ideally GitHub.

Opening files while they are being written

Postby barseno » Mon Jul 18, 2011 1:47 pm

Hi,
I'd like to write an application that performs image processing on very large stacks while they are being acquired. For that I would need to be able to access the file while it is being written. From my tests it seems as if bioformats can't do this right now. Can you think of an easy way to implement that or do you think it's not feasible at all? Help is very appreciated. Thanks!
barseno
 
Posts: 5
Joined: Mon Jul 18, 2011 1:44 pm

Re: Opening files while they are being written

Postby wmoore » Mon Jul 18, 2011 2:06 pm

Hi,

I believe the solution that some have used for this is to write single-plain images as separate files during acquisition & analysis. Then stitch together when you are done.

Will
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: Opening files while they are being written

Postby barseno » Mon Jul 18, 2011 3:22 pm

Thanks for your quick reply. Unfortunately I have no influence on the acquisition software (and it's supposed to be compatible with as many as possible).
barseno
 
Posts: 5
Joined: Mon Jul 18, 2011 1:44 pm

Re: Opening files while they are being written

Postby wmoore » Mon Jul 18, 2011 10:03 pm

Could you tell us what the acquisition software is and what the file format is?

Then I'll have to pass this over to Melissa, who has a better knowledge of file formats than I do.

Thanks,

Will.
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: Opening files while they are being written

Postby mlinkert » Tue Jul 19, 2011 3:54 am

Hi,

I'd like to write an application that performs image processing on very large stacks while they are being acquired. For that I would need to be able to access the file while it is being written. From my tests it seems as if bioformats can't do this right now.


Whether or not this is possible depends entirely upon the format. If all of the metadata (width and height of images, number of total images, etc.) is written first, e.g. into some sort of header file, then it's likely possible.

However, in order to help you make this work, we will need as much information as possible. Can you send a few datasets, preferably in various stages of acquisition? Can you send any of the test code that you are using, so that we know exactly what your code is expecting? Would you rather have Bio-Formats report the number of expected planes (regardless of how many have been acquired already), or only the number of planes that have been acquired so far?

If needed, I can provide you with an SFTP server to which datasets can be uploaded.

Regards,
-Melissa
User avatar
mlinkert
Team Member
 
Posts: 353
Joined: Fri May 29, 2009 2:12 pm
Location: Southwest Wisconsin

Re: Opening files while they are being written

Postby barseno » Thu Jul 28, 2011 11:08 am

Hi,

thanks for your help.

Can you send any of the test code that you are using, so that we know exactly what your code is expecting?

The structure is very simple, like this (in pseudo code):
Code: Select all
while(acquisitionRunning) // tested e.g. by change of file size within certain interval
    new framePackage
    while(framePackage.framesMissing > 0)
        readToPackage(min(framesAvailable,framePackage.framesMissing))
        if(acquisitionRunning == false)
            framePackage.framesMissing = 0
        if(framePackage.framesMissing == 0)
            process(framePackage)


Would you rather have Bio-Formats report the number of expected planes (regardless of how many have been acquired already), or only the number of planes that have been acquired so far?

As seen in the code above I do only care about the frames (/planes) already acquired. However, I will need the frames in (acquisition-) order!

Could you tell us what the acquisition software is and what the file format is?

As I said, I hope it to be working with as many acquisition programs as possible. For now the most important one would be Nikon's ND2 format.

From my understanding it should be possible to achieve it in theory, as long as the file format specifies plane information (as in size & type) in the beginning of the file and then adds new planes in order. Also, it would not work if it used compression that is not being done frame wise (although it sounds weird to me do not do it frame wise, I don't know what's common though).
barseno
 
Posts: 5
Joined: Mon Jul 18, 2011 1:44 pm

Re: Opening files while they are being written

Postby mlinkert » Tue Aug 02, 2011 8:46 pm

Hi,

As seen in the code above I do only care about the frames (/planes) already acquired. However, I will need the frames in (acquisition-) order!


That shouldn't be a problem then.

As I said, I hope it to be working with as many acquisition programs as possible. For now the most important one would be Nikon's ND2 format.

From my understanding it should be possible to achieve it in theory, as long as the file format specifies plane information (as in size & type) in the beginning of the file and then adds new planes in order.


For ND2 specifically, this should mostly work. You may run into some trouble if you need to know the specific timepoint/Z section/channel/etc. of a particular plane, but I think it would work to just retrieve the pixel data.

However, we're not really set up to test this ourselves. If you can send a set of files in various stages of acquisition, that would help a lot in testing. If that's not possible, then it would help to see the full text of any error messages that you are receiving. I will send information privately on how to send files to us.

Also, it would not work if it used compression that is not being done frame wise (although it sounds weird to me do not do it frame wise, I don't know what's common though).


Frame-by-frame compression is the most common approach. In some cases (e.g. ICS) you will see the entire set of pixel data compressed at once; ICS is rarely (never?) used as an acquisition format, though.

The one somewhat common case that I can think of that really won't work at all is if the file is an MS OLE document; Zeiss ZVI, Hamamatsu/SimplePCI .cxd, and Olympus FV1000 .oib all fall into this category.

Regards,
-Melissa
User avatar
mlinkert
Team Member
 
Posts: 353
Joined: Fri May 29, 2009 2:12 pm
Location: Southwest Wisconsin


Return to User Discussion [Legacy]

Who is online

Users browsing this forum: No registered users and 0 guests