Dear Michael,
Can you tell us a little more about your
in-
place-alike import (even share code fragments?): are the files still being slowly brought over to the server at upload time or is the problem when the server processes the files
in later import steps? It does have to do a substantial amount of reading at some point of course. Given your efforts to date you may find the code
in https://gitlab.com/openmicroscopy/incub ... n-importer interesting.
Have you experimented with the parallelization options as at
https://docs.openmicroscopy.org/latest/ ... lel-import? Also, have you studied the import time breakdown? Using OMERO.cli, for image 1234, you can find its fileset ID with,
- Code: Select all
bin/omero obj get Image:1234 fileset
then its import time with,
- Code: Select all
bin/omero fs importtime 567
We do skip import steps when importing publication data submissions to
http://idr.openmicroscopy.org/. For those we use the kind of approach discussed at
https://docs.openmicroscopy.org/latest/ ... -bulk.html: an example configuration for importing one of those datasets
skipping all the deferrable steps is presently at
https://github.com/IDR/idr0053-faas-vir ... A-bulk.yml.
Bio-Formats 6 is to include some exploratory work on importing remotely hosted files such as from Amazon S3. Depending on what you are trying, it might be worth discussing more about that angle?
Cheers,
Mark