We're Hiring!

Using Omero with S3 File System

General and open developer discussion about using OMERO APIs from C++, Java, Python, Matlab and more! Please new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

If you are having trouble with custom code, please provide a link to a public repository, ideally GitHub.

Using Omero with S3 File System

Postby pmann » Fri Mar 08, 2019 4:03 pm

We are trying out Omero, but the images we want to host are quite large (~500 Mb each). We would like to host the images on S3, but Omero requires a file-system to store the images in. Would it be possible to use something like s3fs to link the folder where Omero stores the images to S3? If not, them I'm curious about what other people have done to get around this. Is the solution just to get more disc storage?

Thanks for the help.
pmann
 
Posts: 3
Joined: Thu Mar 07, 2019 4:55 pm

Re: Using Omero with S3 File System

Postby manics » Fri Mar 08, 2019 5:03 pm

Hi

Officially OMERO only supports POSIX file systems. The Image Data Repository has around 90TB of files including several multi-gigabyte images so size isn't a problem as far as OMERO is concerned.

BioFormats 6.0.0 adds preliminary support for reading off S3. For instance if you download the BioFormats command line tools you can use the showinf command to open an S3 image:
Code: Select all
./bftools/showinf s3://s3.amazonaws.com/bucket-name/path/to/image.jpg

If you try this one obvious issue is the latency in opening and viewing images. There's an optional cache included in the S3 reader but that's not a full solution, for example it doesn't automatically refresh, nor is there a size limit.

Getting this into OMERO will require a bit of work as the current release 5.4.10 uses BioFormats 5.9.2.
In addition you'll need to compile and include this reader which is a wrapper to store the URL https://gitlab.com/openmicroscopy/incub ... urlreader/

In the current architecture OMERO always requires a local file on disk, which is provided by this wrapper that stores the URL and some options. The S3 file is then streamed remotely.

I've got an OMERO.server built with an older version of the S3 work running, so we know it's possible. It's definitely not suitable for production use. Supporting object stores in OMERO is in our long term plans, so if you're interested in trying it out and giving feedback on how it should work your input would be very welcome!

I haven't tried S3fs, but I suspect you'd run into the same problems with latency. If you're also writing to S3fs through OMERO it's possible you'll have problems with file locks. If you try it I'm sure others would be interested to know whether it works.

Simon
User avatar
manics
Team Member
 
Posts: 261
Joined: Mon Oct 08, 2012 11:01 am
Location: Dundee

Re: Using Omero with S3 File System

Postby pmann » Fri Mar 08, 2019 7:18 pm

Hey Simon,

Thank you for responding to my question. It's great that S3 support is a long term goal for Omero.

If the S3FS idea doesn't work out, then another option we're thinking about is to archive images into S3 and then delete them from Omero after a regular amount of time or when our capacity gets filled up.

I'll post here again if the S3FS solution performs well but I'm expecting that it will cause a lot of problems.

Thanks again,
Philip
pmann
 
Posts: 3
Joined: Thu Mar 07, 2019 4:55 pm


Return to Developer Discussion

Who is online

Users browsing this forum: No registered users and 1 guest