We're Hiring!

Import with python problems

General and open developer discussion about using OMERO APIs from C++, Java, Python, Matlab and more! Please new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

If you are having trouble with custom code, please provide a link to a public repository, ideally GitHub.

Import with python problems

Postby sukunis » Thu Jun 14, 2018 8:24 am

Hallo everybody,

I will import a multiframe single channel uint16 *.ome.tif file with python after an image processing step on the server.
But after the import the histogram of the image seems to be stretched.
Import the same file with the OMERO.importer everything is fine...
Here is what I do in python:
Code: Select all
...
def plane_generator():
       img = Image.open(imagePath)
       for frame in ImageSequence.Iterator(img):
           frame.load()
           numpyPlane = numpy.array(frame)
           yield numpyPlane

plane_gen = plane_generator()
newImg = conn.createImageFromNumpySeq(
        plane_gen, imageName, sizeZ=21,sizeC=1,sizeT=21, description='Test',dataset=imgSrc.getParent())
...


Does somebody has any idea ?

Best Regards
Susanne
sukunis
 
Posts: 16
Joined: Fri Sep 18, 2015 8:12 am

Re: Import with python problems

Postby wmoore » Fri Jun 15, 2018 3:55 pm

Hi Suzanne,

Apologies for the delay.
I have reproduced your problem with the uint16 image import, but we haven't figured out exactly what's going on or have a fix yet.

It occurred to me that one option is to use Python to create a FileSet on the server, upload the files and trigger import on the server.
I have some code to do that at https://gist.github.com/will-moore/f540 ... 8b38a077ce but it could probably do with some cleaning up.

I'll try and discuss with Josh and and work out what's best,

Regards,

Will.
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: Import with python problems

Postby jmoore » Mon Jun 18, 2018 10:04 am

Hi Susanne,

what was the original reason for not using the importer in the first place? I assume something like that you're trying to put this in a larger pipeline. Can you describe the whole workflow to us so we can make an alternative suggestion (while we try to figure out the uint16 bug)?

~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Import with python problems

Postby sukunis » Tue Jun 19, 2018 7:17 am

Hi Josh,

here is my desired workflow:
- start headless imagej image processing on server by OMERO.script for selected images
- save results back to OMERO as new images

Regards,
Susanne
sukunis
 
Posts: 16
Joined: Fri Sep 18, 2015 8:12 am

Re: Import with python problems

Postby wmoore » Tue Jun 19, 2018 9:09 pm

Hi Susanne,

I have created an example script to demo how to import from a Python script by using the CLI import.

https://gitlab.com/openmicroscopy/incub ... _import.py

Hopefully that approach will work for you.

Regards,

Will.
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: Import with python problems

Postby wmoore » Tue Jun 19, 2018 9:13 pm

Hi Susanne,

Just something to be aware of when running ImageJ from Python scripts on the server: If you have multiple users calling ImageJ at once you may see some concurrency issues since this is not a use-case that IJ was designed for.

Regards,

Will.
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: Import with python problems

Postby wmoore » Tue Jun 19, 2018 11:15 pm

I just saw Josh's suggestion at viewtopic.php?f=6&t=8449#p19535 of doing this, which doesn't need to set the "path/to/bin/omero" as in my example script.

Code: Select all
from omero.cli import cli_login
with cli_login() as cli:
    cli.invoke("import --help")


So this approach might be better (but I haven't tested this in an OMERO.script yet).
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: Import with python problems

Postby jmoore » Wed Jun 20, 2018 5:37 am

The same currently won't work for scripts because of the need to use the `omero.scripts.client` method, but something fairly similar will work in a script:

Code: Select all
import omero
import omero.cli
import omero.scripts

client = omero.scripts.client("test")
cli = omero.cli.CLI()
cli.loadplugins()
cli.set_client(client)
cli.invoke("hql -q 'select g.name from ExperimenterGroup g'")


produces:

Code: Select all
$ bin/omero script launch /susanne.py
Job 102126 ready
Waiting....
Callback received: FINISHED

   *** start stdout (id=193877)***
   *  #  | Col1
   * ----+------------------------------------------------------------
   *  0  | system
   *  1  | user
   *  2  | guest
        * ...
   * (25 rows)
   *
   *** end stdout ***


   *** out parameters ***
   ***  done ***


It shouldn't be too hard to provide an `omero.scripts.cli` method which combines the two.

~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Import with python problems

Postby wmoore » Wed Jun 20, 2018 10:55 pm

Hi Josh,

I tried to take what you provided and use it in the script I posted above to import (see below).

However, the stderr I get seems to indicate the import hasn't worked.

Code: Select all
2018-06-20 23:49:29,034 216        [      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.4.7-ice36-SNAPSHOT
2018-06-20 23:49:29,045 227        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 5.8.2 revision: 07ecd9a5d239985f8ae57462c12124c2fa5ccd9b date: 18 April 2018
2018-06-20 23:49:29,075 257        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2018-06-20 23:49:29,349 531        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2018-06-20 23:49:29,396 578        [      main] INFO      ome.formats.importer.ImportCandidates - 1 file(s) parsed into 1 group(s) with 1 call(s) to setId in 45ms. (47ms total) [0 unknowns]

Usage:  importer-cli [OPTION]... [path [path ...]]...
   or:   importer-cli [OPTION]... -
...


Also, when I try to
Code: Select all
client.setOutput("Message", rstring("Import complete"))
at the end
of the script, I see
Code: Select all
omero.ClientError: No session available
in stderr, suggesting that
using the CLI has closed my session?

Script (run it on a Dataset)
Code: Select all

import omero.scripts as scripts
from omero.gateway import BlitzGateway
import omero
import omero.cli

from omero.rtypes import rlong, rstring


def import_image(conn, scriptParams):

    dataset = conn.getObject("Dataset", scriptParams["IDs"][0])

    # Create test image to import
    image = list(dataset.listChildren())[0]
    pil_image = image.renderImage(0, 0)
    image_name = "test_import.png"
    pil_image.save(image_name)

    # import via cli
    client = conn.c
    cmd = "import %s -d %s" % (image_name, dataset.id)
    print cmd

    cli = omero.cli.CLI()
    cli.loadplugins()
    cli.set_client(client)
    cli.invoke(cmd)


def runAsScript():
    """
    The main entry point of the script, as called by the client via the
    scripting service, passing the required parameters.
    """

    dataTypes = [rstring('Dataset')]

    client = scripts.client(
        'Import_CLI_Test.py',
        """Testing import via CLI from script""",

        scripts.String(
            "Data_Type", optional=False, grouping="1",
            description="Choose source of images (only Dataset supported)",
            values=dataTypes, default="Dataset"),

        scripts.List(
            "IDs", optional=False, grouping="2",
            description="Dataset ID").ofType(rlong(0)),
    )

    try:
        scriptParams = client.getInputs(unwrap=True)
        print scriptParams
        conn = BlitzGateway(client_obj=client)
        import_image(conn, scriptParams)

        client.setOutput("Message", rstring("Import complete"))
           
    finally:
        client.closeSession()

if __name__ == "__main__":
    runAsScript()
User avatar
wmoore
Team Member
 
Posts: 674
Joined: Mon May 18, 2009 12:46 pm

Re: Import with python problems

Postby jmoore » Thu Jun 21, 2018 6:06 am

Sorry, I should have written up a simple complete example. Here the primary difference is the use of `invoke("login")` rather than `set_client`:

Code: Select all
import omero.scripts as scripts
from omero.gateway import BlitzGateway
import omero
import omero.cli

from omero.rtypes import rlong, rstring


def import_image(conn, scriptParams):

    dataset = conn.getObject("Dataset", scriptParams["IDs"][0])

    # Create test image to import
    image = list(dataset.listChildren())[0]
    pil_image = image.renderImage(0, 0)
    image_name = "test_import.png"
    pil_image.save(image_name)

    # import via cli
    client = conn.c
    cmd = "import %s -d %s" % (image_name, dataset.id)
    print cmd

    cli = omero.cli.CLI()
    cli.loadplugins()
    cli.set_client(client)
    cli.invoke(cmd)


def runAsScript():
    """
    The main entry point of the script, as called by the client via the
    scripting service, passing the required parameters.
    """

    dataTypes = [rstring('Dataset')]

    client = scripts.client(
        'Import_CLI_Test.py',
        """Testing import via CLI from script""",

        scripts.String(
            "Data_Type", optional=False, grouping="1",
            description="Choose source of images (only Dataset supported)",
            values=dataTypes, default="Dataset"),

        scripts.List(
            "IDs", optional=False, grouping="2",
            description="Dataset ID").ofType(rlong(0)),
    )

    try:
        scriptParams = client.getInputs(unwrap=True)
        print scriptParams
        conn = BlitzGateway(client_obj=client)
        import_image(conn, scriptParams)

        client.setOutput("Message", rstring("Import complete"))
           
    finally:
        client.closeSession()

if __name__ == "__main__":
    runAsScript()


The same change to your script leads to:

Code: Select all
Enter value for "IDs": 24501
Job 102246 ready
Waiting....
Callback received: FINISHED

   *** start stdout (id=193959)***
   * Image:131203
   * {'IDs': [24501L], 'Data_Type': 'Dataset'}
   * import test_import.png -d 24501
   *
   *** end stdout ***


   *** start stderr (id=193960)***
   * Using session 07c63559-072a-4104-b2f9-53cc30a29c0f (root@eel.openmicroscopy.org:4064). Idle timeout: 10 min. Current group: read-write-1
   * Using session 07c63559-072a-4104-b2f9-53cc30a29c0f (root@eel.openmicroscopy.org:4064). Idle timeout: 10 min. Current group: read-write-1
   * 2018-06-21 07:06:15,170 247        [      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.4.6-105-98d0541-ice35-b1004
   * 2018-06-21 07:06:15,185 262        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 5.8.2 revision: 6bc10be63e934ed12e76c3ffd83ecd2bca6312a1 date: 18 April 2018
   * 2018-06-21 07:06:15,223 300        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
   * 2018-06-21 07:06:15,587 664        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
   * 2018-06-21 07:06:15,642 719        [      main] INFO      ome.formats.importer.ImportCandidates - 1 file(s) parsed into 1 group(s) with 1 call(s) to setId in 51ms. (54ms total) [0 unknowns]
   * 2018-06-21 07:06:15,778 855        [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to eel.openmicroscopy.org:4064
   * 2018-06-21 07:06:16,356 1433       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
   * 2018-06-21 07:06:16,757 1834       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.4.6
   * 2018-06-21 07:06:16,757 1834       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.4.6-105-98d0541-ice35-b1004
   * 2018-06-21 07:06:16,757 1834       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 1.7.0_80
   * 2018-06-21 07:06:16,757 1834       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
   * 2018-06-21 07:06:16,757 1834       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
   * 2018-06-21 07:06:16,757 1834       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 2.6.32-696.6.3.el6.x86_64
   * 2018-06-21 07:06:17,142 2219       [      main] INFO          ome.formats.importer.ImportConfig - Using import target: Dataset:24501
   * 2018-06-21 07:06:17,167 2244       [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
   * 2018-06-21 07:06:17,919 2996       [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
   * 2018-06-21 07:06:17,930 3007       [      main] INFO   ts.importer.transfers.UploadFileTransfer - Transferring /repositories/tmp/omero_hudson/15523/processK_D5q1.dir/test_import.png...
   * 2018-06-21 07:06:17,963 3040       [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /repositories/tmp/omero_hudson/15523/processK_D5q1.dir/test_import.png
   * 2018-06-21 07:06:18,078 3155       [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /repositories/tmp/omero_hudson/15523/processK_D5q1.dir/test_import.png
   * 2018-06-21 07:06:18,211 3288       [      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
   * 2018-06-21 07:06:18,308 3385       [      main] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 193957
   * 2018-06-21 07:06:18,656 3733       [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5  Logfile: 193957
   * 2018-06-21 07:06:19,050 4127       [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 193957
   * 2018-06-21 07:06:19,508 4585       [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5  Logfile: 193957
   * 2018-06-21 07:06:19,550 4627       [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 193957
   * 2018-06-21 07:06:19,592 4669       [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 193957
   * 2018-06-21 07:06:19,764 4841       [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /repositories/tmp/omero_hudson/15523/processK_D5q1.dir/test_import.png
   * Other imported objects:
   * Fileset:32003
   * 2018-06-21 07:06:19,766 4843       [l.Client-0] INFO      ome.formats.importer.cli.ErrorHandler - Number of errors: 0
   *
   * ==> Summary
   * 1 file uploaded, 1 fileset created, 1 image imported, 0 errors in 0:00:02.905
   *
   *** end stderr ***


   *** out parameters ***
   * Message=Import complete
   ***  done ***
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Next

Return to Developer Discussion

Who is online

Users browsing this forum: No registered users and 0 guests