We're Hiring!

Downloading multiple files from OMERO.web is limited to 2GiB

General user discussion about using the OMERO platform to its fullest. Please ask new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

There are workflow guides for various OMERO functions on our help site - http://help.openmicroscopy.org

You should find answers to any basic questions about using the clients there.

Downloading multiple files from OMERO.web is limited to 2GiB

Postby Sethur » Thu Jul 30, 2015 11:59 am

Hi,

some of our users recently wanted to download rather larger amounts of data in multiple files that together exceeded 2 GiB. This will currently fail in OMERO.web because of the 2 GiB file size limit for ZIP files. A solution would be to use 7z or another more modern ZIP format as a second option.

Cheers,

Tristan
Sethur
 
Posts: 112
Joined: Thu Jan 16, 2014 11:34 pm

Re: Downloading multiple files from OMERO.web is limited to

Postby wmoore » Thu Jul 30, 2015 12:13 pm

Hi Tristan,

Thanks for the feedback and suggestion.
I don't know if you're using OMERO 5.1.2 (or later) but this added support for downloading individual files from web (see https://github.com/openmicroscopy/openm ... /pull/3785).

So, this would allow you to pick individual files, even if the total size of multiple zipped files was > 2 GiB.
This also removes the overhead of creating a zip in the web server, since the files are streamed directly from OMERO so there shouldn't be a file size limit.

I hope that helps?

Cheers,

Will.
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: Downloading multiple files from OMERO.web is limited to

Postby manics » Thu Jul 30, 2015 1:11 pm

Hi Tristan

The Python ZipFile module can be changed to support large files (see the allowZip64 flag https://docs.python.org/2/library/zipfi ... le-objects) but as Will pointed out it imposes a large overhead on the server.

Simon
User avatar
manics
Team Member
 
Posts: 261
Joined: Mon Oct 08, 2012 11:01 am
Location: Dundee

Re: Downloading multiple files from OMERO.web is limited to

Postby Sethur » Sun Aug 02, 2015 9:56 pm

Hi Will & Simon,

thanks for you quick response. We are using 5.1.3, and yes, downloading individual files from the dialog is certainly a way around the 2 GiB limit. I agree that implementing the option of having extremely large ZIP files generated on the server side would have the potential of creating unsustainable overhead, so the limit is probably a good idea. However, the user needs to be aware of this limitation. There cannot be an option of creating a zip file if the predicted final file size is over 2 GB. Instead, there should be an option of splitting the download into a set of ZIP files (<= 2GiB) and raw data (potentially > 2GiB). These zip files should only be generated the moment the user actually requests the download to limit the maximum overhead.

Also: Multiple file uploads are no problem with current HTML versions, so is there any progress on importing via OMERO.web? I would really like to tell our users that there is no reason any more to use the Java client. Currently, manual upload, ROIs and multiple downloads is the only reason to use Insight.

Cheers,

Tristan
Sethur
 
Posts: 112
Joined: Thu Jan 16, 2014 11:34 pm

Re: Downloading multiple files from OMERO.web is limited to

Postby wmoore » Tue Aug 04, 2015 12:07 pm

Hi Tristan,

Zip files are currently generated 'on the fly' when the user requests the download, but clearly the user shouldn't be able to request this if the size is over 2 GB.
Thanks for your suggestions on improving the download.
Ticket created: https://trac.openmicroscopy.org/ome/ticket/12980

I'm afraid we are not currently working on web-based import, although we are discussing our next steps for supporting ROI creation/editing in web.

Regards,

Will.
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: Downloading multiple files from OMERO.web is limited to

Postby PhilippeP » Mon Jan 11, 2016 1:18 pm

A question that I guess is in the same vein:
We generate a lot of small files (segmented cells and associated data) following image processing in a collaboration with other labs. We get (tens of) thousands of files under multiple formats, and we would like to associate these files to our stacks on our OMERO server. Since the files are small but quite numerous, we zip them. Unfortunately, using OMERO Insight 5.2.0 ice35 b12, uploading for example a 4.2Go Zip always fail.
Is there an alternative way for us to upload this Zip "next" to our data so our colleagues can easily find them?
Thanks for your time.
Philippe
PhilippeP
 
Posts: 44
Joined: Tue Oct 22, 2013 1:31 pm

Re: Downloading multiple files from OMERO.web is limited to

Postby Dominik » Tue Jan 12, 2016 10:56 am

Hi Philippe,

just to clarify: It is not a zip file with images, it's a zip file with various data, and you're trying to upload it as file attachment (not via the importer), right?

Kind Regards,
Dominik
User avatar
Dominik
Team Member
 
Posts: 149
Joined: Mon Feb 10, 2014 11:26 am

Re: Downloading multiple files from OMERO.web is limited to

Postby PhilippeP » Wed Jan 13, 2016 2:56 pm

Oops...
Thank you Dominik for your enlightening response. I was indeed out to lunch, trying to upload the zip the "classical" way, via the importer...
Uploading zip files as an attachment does work perfectly.
:oops:
PhilippeP
 
Posts: 44
Joined: Tue Oct 22, 2013 1:31 pm

Re: Downloading multiple files from OMERO.web is limited to

Postby Dominik » Wed Jan 13, 2016 4:21 pm

Ok, good to hear. The importer tries to interpret everything as image file.

Regards,
Dominik
User avatar
Dominik
Team Member
 
Posts: 149
Joined: Mon Feb 10, 2014 11:26 am


Return to User Discussion

Who is online

Users browsing this forum: No registered users and 1 guest