We're Hiring!

Batch export with Python

General and open developer discussion about using OMERO APIs from C++, Java, Python, Matlab and more! Please new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

If you are having trouble with custom code, please provide a link to a public repository, ideally GitHub.

Batch export with Python

Postby jstitlow » Thu Jun 14, 2018 10:43 am

Is there a way to call the CLI export command from Python?

Or more to the point, after retrieving all of the images from a dataset with the code below, how do I save them in their original format (or OME.tiff)?

Many thanks!
j

Code: Select all
import omero
from omero.gateway import BlitzGateway
import os
import getpass

USER = os.environ['USER']
PASS = getpass.getpass("Enter Password:")
HOST = os.environ['HOST']

conn = BlitzGateway(USER, PASS, host=HOST, port=4064)
conn.connect()
conn.getSession().setTimeToIdle(rlong(60*60*1000))

DATASET = 16822

dataset = conn.getObject('Dataset', DATASET)

# make a dict of name: image
images = {}
for image in dataset.listChildren():
        filename = image.getName()
        print filename
jstitlow
 
Posts: 18
Joined: Sat Feb 03, 2018 9:07 pm

Re: Batch export with Python

Postby wmoore » Fri Jun 15, 2018 3:44 pm

Hi Josh

This should do what you want.
NB: it assumes that there are not duplicate file names (will get overwritten)

Code: Select all
import omero
from omero.gateway import BlitzGateway

conn = BlitzGateway("username", "password", host="localhost", port=4064)
conn.connect()
conn.SERVICE_OPTS.setOmeroGroup(-1)

dataset_id = 11509

for image in conn.getObject("Dataset", dataset_id).listChildren():

  for orig_file in image.getImportedImageFiles():

    file_name = orig_file.getName()
    print "Downloading...", file_name

    with open(file_name, "wb") as f:
      for chunk in orig_file.getFileInChunks(buf=2621440):
        f.write(chunk)

conn.close()


Hope that works for you,

Will.
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: Batch export with Python

Postby wmoore » Fri Jun 15, 2018 3:46 pm

If you want to be able to run this from the command line, passing in username, password, dataset_id, you can follow the pattern at e.g. https://github.com/ome/training-scripts ... te_ROIs.py
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: Batch export with Python

Postby wmoore » Fri Jun 15, 2018 10:02 pm

If you want to download images linked to Tags, via command-line, save this code as "download_original_files_by_tag.py"

Code: Select all
import argparse
from omero.gateway import BlitzGateway

def run(username, password, tag_id, host, port):
    conn = BlitzGateway(username, password, host=host, port=port)
    try:
        conn.connect()
        for link in conn.getAnnotationLinks("Image", ann_ids=[tag_id]):
            image = link.getParent()
            print 'Image', image.getName()
            for orig_file in image.getImportedImageFiles():
                file_name = orig_file.getName()
                print " Downloading...", file_name

                with open(file_name, "wb") as f:
                  for chunk in orig_file.getFileInChunks(buf=2621440):
                    f.write(chunk)
           
    except Exception as exc:
            print "Error while deleting annotations: %s" % str(exc)
    finally:
        conn.close()

def main(args):
    parser = argparse.ArgumentParser()
    parser.add_argument('username')
    parser.add_argument('password')
    parser.add_argument('tag_id')
    parser.add_argument('--server', help="OMERO server hostname")
    parser.add_argument('--port', default=4064, help="OMERO server port")
    args = parser.parse_args(args)
    run(args.username, args.password, args.tag_id, args.server, args.port)

if __name__ == '__main__':
    import sys
    main(sys.argv[1:])


Then on the command-line you can do:
Code: Select all
$ python download_original_files_by_tag.py username password 44409 --server server.address
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: Batch export with Python

Postby jstitlow » Mon Jun 18, 2018 9:25 am

That's perfect, thank you Will!
jstitlow
 
Posts: 18
Joined: Sat Feb 03, 2018 9:07 pm

Re: Batch export with Python

Postby jstitlow » Tue Jun 19, 2018 7:36 pm

Hi Will,

The export code gives an error (below) when exporting images that were created from a fileset, e.g., .mvd2.

Is there a way around this?

Code: Select all
Traceback (most recent call last):
  File "/usr/people/bioc1301/src/OMERO_scripts/OMERO_export_dataset_CLI.py", line 20, in <module>
    for image in dataset.listChildren():
AttributeError: 'NoneType' object has no attribute 'listChildren'
jstitlow
 
Posts: 18
Joined: Sat Feb 03, 2018 9:07 pm

Re: Batch export with Python

Postby jstitlow » Tue Jun 19, 2018 8:24 pm

Removing .list.children(), i.e,

Code: Select all
#for image in dataset.listChildren():
for image in dataset:


shows that the dataset is not iterable:

Code: Select all
Traceback (most recent call last):
  File "/usr/people/bioc1301/src/OMERO_scripts/OMERO_export_dataset_CLI.py", line 21, in <module>
    for image in dataset:
TypeError: 'NoneType' object is not iterable
jstitlow
 
Posts: 18
Joined: Sat Feb 03, 2018 9:07 pm

Re: Batch export with Python

Postby jmoore » Tue Jun 19, 2018 8:46 pm

Code: Select all
AttributeError: 'NoneType' object has no attribute 'listChildren'


Here NoneType means that your original call, "dataset = conn.getObject('Dataset', DATASET)", didn't return anything. We need to back up to that point for now. Can you double check the dataset ID? Does the user your logging in as have permission to view it?

~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Batch export with Python

Postby jstitlow » Tue Jun 19, 2018 9:33 pm

Permissions might be the problem as I am now experiencing other password issues, let me get back to you on this...
Last edited by jstitlow on Tue Jun 19, 2018 10:02 pm, edited 1 time in total.
jstitlow
 
Posts: 18
Joined: Sat Feb 03, 2018 9:07 pm

Re: Batch export with Python

Postby jmoore » Tue Jun 19, 2018 9:43 pm

You might need to check the server log for a resource exhaustion of some form.
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Next

Return to Developer Discussion

Who is online

Users browsing this forum: No registered users and 0 guests