OME-NGFF 2021 updates

back to Blog

In 2020, we published two blog posts reporting on the development of next-generation file formats (NGFFs). We announced public specifications and shared public examples of multiscale images and label images in November 2020 and high-content screening (HCS) datasets in December 2020.

Here we report progress made on OME-NGFF over the course of 2021 with contributions from OME, Glencoe Software, EMBL and Harvard Medical School.

OME-NGFF paper

The principle of next-generation file formats as a solution for bioimaging data storage and access was recently published as a Brief Communication in Nature Methods. This work is driven by use cases in image data publication and domains with large bioimaging datasets– public data resources, high content screening and light sheet microscopy with direct contributions from all the groups mentioned above. We have tried to make the work as widely applicable as possible, while trying to deliver examples of real world solutions.

This paper includes a benchmark which measures the access times for different types of data chunks. The reading performance of NGFFs is compared to the established TIFF and HDF5 format for different imaging modalities in different types of storage. These results demonstrate the benefits of the different formats under different access scenarios, e.g., on a local computer, in the cloud, etc.. We discuss the advantages and tradeoffs of each format in various contexts. We believe there is no single format that provides optimal performance in all scenarios. The corollary to this is that any imaging data-intensive project will have to consider these issues and make informed choices regarding the data structures they use. See the “Outlook” section of the paper for more details.

Dimension separator

Preliminary results for the latency benchmark of the OME-NGFF paper revealed performance issues when accessing Zarr chunks remotely for some modalities. Our tests showed that the source of these issues was related to the separator used between chunks in the Zarr format. The version 0.2 includes a backwards incompatible change to use slash (/) as the separator between individual chunks rather than dot (.). This separator must be used for all multiscale images of all modalities, including label images and high-content screening fields of views. Moving forward, this restriction will be eased, but the underlying libraries like Zarr will be moving to / as the default.

2D-5D axes

Until version 0.2, the axes of multiscales images were implicitly assumed to be XYZCT. The 0.3 version of the OME-NGFF specification loosened this requirement by introducing a mandatory axes attribute in the multiscales specification. This extends dimensionality between 2D and 5D. For instance, it is possible to store a time-lapse 2D image using XYT or a three-dimensional volume as XYZ.

OME-NGFF sample data

The two features discussed above (dimension separator and 2D-5D axes) are published in the latest OME-NGFF specification, currently at version 0.3. We generated a comprehensive set of 0.3 OME-NGFF samples to cover all the current set of specifications:

Image Study Axes Dimensions Others Viewer S3-endpoint
IDR thumbnail for image:9836998 idr0079 XYZC 1584 x 788 x 142 x 2 labels (0) view
IDR thumbnail for image:5514375 idr0052 XYZCT 256 x 256 x 31 x 3 x 40 labels (cells, chromosomes) view
IDR thumbnail for image:11511419 idr0095 XYC 2048 x 2048 x 3 labels (0) view
IDR thumbnail for image:11511420 idr0095 XYC 2048 x 2048 x 3 labels (0) view
IDR thumbnail for image:11511421 idr0095 XYC 2048 x 2048 x 3 labels (0) view
IDR thumbnail for image:11511422 idr0095 XYC 2048 x 2048 x 3 labels (0) view
IDR thumbnail for image:11511423 idr0095 XYC 2048 x 2048 x 3 labels (0) view
IDR thumbnail for image:11511424 idr0095 XYC 2048 x 2048 x 3 labels (0) view
IDR thumbnail for image:12922361 idr0109 XYT 2560 x 2160 x 721 view
IDR thumbnail for image:9528933 idr0075 XYZ 512 x 512 x 91 view
IDR thumbnail for image:4007817 idr0051 XYZT 333 x 333 x 201 x 79 view
IDR thumbnail for image:3491626 idr0040 XYCT 2048 x 2048 x 5 x 20 view
IDR thumbnail for image:10503601 idr0094 XY 1080 x 1080 96 wells view

December 16, 2021

back to top