We're Hiring!

image import python script

General and open developer discussion about using OMERO APIs from C++, Java, Python, Matlab and more! Please new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

If you are having trouble with custom code, please provide a link to a public repository, ideally GitHub.

image import python script

Postby bhcho » Tue Nov 08, 2011 7:05 pm

Hi all,

I want to import lots of images by python script.

1.
Could anyone show me a snippet of how to import images by python script?

2.
Is it possible to specify the dataset ID when I import an image using the script?

3.
does this way to import images calls my custom Bridge (extensions.jar)?

Best,
BK
bhcho
 
Posts: 236
Joined: Mon Apr 05, 2010 2:15 pm

Re: image import python script

Postby icaoberg » Wed Nov 09, 2011 3:19 pm

Dr. Cho

1. To import an image from omero import use the following line
Code: Select all
omero -s localhost -u "omero" -w "omero" -p 4064 import -d 1601 001_1_001.ome.tif


2. In the above example the option -d 1601 tells the code to import it to dataset 1601

3. It calls the same bridge.

Ivan

bhcho wrote:Hi all,

I want to import lots of images by python script.

1.
Could anyone show me a snippet of how to import images by python script?

2.
Is it possible to specify the dataset ID when I import an image using the script?

3.
does this way to import images calls my custom Bridge (extensions.jar)?

Best,
BK
icaoberg
 
Posts: 145
Joined: Fri Sep 17, 2010 9:05 pm
Location: Pittsburgh, PA

Re: image import python script

Postby bhcho » Wed Nov 09, 2011 3:29 pm

Thanks Ivan
:D
bhcho
 
Posts: 236
Joined: Mon Apr 05, 2010 2:15 pm

Re: image import python script

Postby bhcho » Wed Nov 09, 2011 4:27 pm

I have follow-up questions.

1.
Does this connection have a timeout?
if no, does it mean that the connection can run forever?
if yes, how long is the default timeout and how can I control it (by another option)?

2.
After several successful trials with the command above, i got the following outofmemory error.
Could anyone tell me why this happens?
Using session 7b5da05e-05d3-454a-8d0e-fa3c5077e26a (demo@localhost). Idle timeout: 10.0 min. Current group: Researchers
2011-11-09 12:16:41,489 0 [ main] INFO ome.formats.importer.ImportConfig - OMERO Version: Beta4.2.1-r8614-Beta4.2-b41
2011-11-09 12:16:42,468 979 [ main] WARN ome.system.UpgradeCheck - UPGRADE AVAILABLE:Please upgrade to Beta-4.3.3 See http://trac.openmicroscopy.org.uk/omero for the latest version
2011-11-09 12:16:42,478 989 [ main] INFO ome.formats.importer.ImportConfig - Bioformats version: 4.2.1-DEV revision: 7178 date: 9 November 2010
2011-11-09 12:16:42,493 1004 [ main] INFO formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2011-11-09 12:16:42,879 1390 [ main] INFO ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2011-11-09 12:16:43,964 2475 [ main] INFO ome.formats.importer.ImportCandidates - 2 file(s) parsed into 2 group(s) with 2 call(s) to setId in 1078ms. (1085ms total) [0 unknowns]
2011-11-09 12:16:44,036 2547 [ main] INFO ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
2011-11-09 12:16:44,753 3264 [ main] INFO ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2011-11-09 12:16:44,793 3304 [ main] ERROR formats.importer.cli.CommandLineImporter - Error during import process.
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
at IceInternal.ThreadPool.<init>(ThreadPool.java:91)
at Ice.ObjectAdapterI.<init>(ObjectAdapterI.java:823)
at IceInternal.ObjectAdapterFactory.createObjectAdapter(ObjectAdapterFactory.java:183)
at Ice.CommunicatorI.createObjectAdapter(CommunicatorI.java:71)
at omero.client.init(client.java:364)
at omero.client.<init>(client.java:255)
at omero.client.createClient(client.java:415)
at ome.formats.OMEROMetadataStoreClient.unsecure(OMEROMetadataStoreClient.java:587)
at ome.formats.OMEROMetadataStoreClient.initialize(OMEROMetadataStoreClient.java:552)
at ome.formats.importer.ImportConfig.createStore(ImportConfig.java:292)
at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:88)
at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:365)



Best,
BK
bhcho
 
Posts: 236
Joined: Mon Apr 05, 2010 2:15 pm

Re: image import python script

Postby bhcho » Sun Nov 20, 2011 2:02 am

I have another type of error when I import images via the script.

Code: Select all
Created session 0a7a9e0a-c2fa-47ae-a428-18d803c41848 (performance@localhost). Idle timeout: 10.0 min. Current group: Researchers
2011-11-19 21:00:05,783 0          [     main] INFO          ome.formats.importer.ImportConfig  - OMERO Version: Beta4.2.1-r8614-Beta4.2-b41
2011-11-19 21:00:09,166 3383      [     main] WARN                    ome.system.UpgradeCheck  - UPGRADE AVAILABLE:Please upgrade to Beta-4.3.3 See http://trac.openmicrosc$
2011-11-19 21:00:09,175 3392      [     main] INFO          ome.formats.importer.ImportConfig  - Bioformats version: 4.2.1-DEV revision: 7178 date: 9 November 2010
2011-11-19 21:00:09,187 3404      [     main] INFO   formats.importer.cli.CommandLineImporter  - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2011-11-19 21:00:09,571 3788      [     main] INFO     ome.formats.importer.ImportCandidates  - Depth: 4 Metadata Level: MINIMUM
2011-11-19 21:00:22,475 16692      [     main] INFO     ome.formats.importer.ImportCandidates  - 400 file(s) parsed into 400 group(s) with 400 call(s) to setId in 12311ms.$
2011-11-19 21:00:22,542 16759      [     main] INFO      ome.formats.OMEROMetadataStoreClient  - Attempting initial SSL connection to localhost:4064
2011-11-19 21:00:23,226 17443      [     main] INFO      ome.formats.OMEROMetadataStoreClient  - Insecure connection requested, falling back
2011-11-19 21:00:23,258 17475      [     main] ERROR  formats.importer.cli.CommandLineImporter  - Error during import process.
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:597)
        at IceInternal.ThreadPool.<init>(ThreadPool.java:91)
        at Ice.ObjectAdapterI.<init>(ObjectAdapterI.java:823)
        at IceInternal.ObjectAdapterFactory.createObjectAdapter(ObjectAdapterFactory.java:183)
        at Ice.CommunicatorI.createObjectAdapter(CommunicatorI.java:71)
        at omero.client.init(client.java:364)
        at omero.client.<init>(client.java:255)
        at omero.client.createClient(client.java:415)
        at ome.formats.OMEROMetadataStoreClient.unsecure(OMEROMetadataStoreClient.java:587)
        at ome.formats.OMEROMetadataStoreClient.initialize(OMEROMetadataStoreClient.java:552)
        at ome.formats.importer.ImportConfig.createStore(ImportConfig.java:292)
        at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:88)
        at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:365)
bhcho
 
Posts: 236
Joined: Mon Apr 05, 2010 2:15 pm

Re: image import python script

Postby jmoore » Mon Nov 21, 2011 8:01 am

Hi BK,

interesting. I've personally never seen an OOM so early in the import process. What JVM are you using and on what operating system? What type of files are these and how large?

To try to get past this point, you'll either need to increase the memory for the process:
Code: Select all
export JAVA_OPTS=-Xmx1000M

or loop over all the files in Python as opposed to passing them all to the importer at once. However, since I'm not sure what's causing the OOM, it's not clear if this will fix the situation. I've filed a ticket, and anything you can tell us about these files would be greatly appreciated.

Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: image import python script

Postby bhcho » Mon Nov 21, 2011 4:00 pm

Hi Josh,

1.
for Java,
java version "1.6.0_16"
Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode)

for OS version,
Linux version 2.6.28.4-002-STD64 (gcc version 4.3.2 20081105 (Red Hat 4.3.2-7) (GCC) ) #1 SMP PREEMPT Fri Jul 3 20:28:31 EDT 2009

2.
I will try to use the JAVA memory option, and let you know if I still have the same problem.

3.
for looping over all the files in Python, do you mean I call the shell script for each file from python?
Code: Select all
omero -s localhost -u "omero" -w "omero" -p 4064 import -d 1601 001_1_001.ome.tif

I think this will create a session everytime I call this script (which might be a problem).
Can you tell me about a python script (not shell script) that imports images like the shell script above, not creating a session for each image (to a dedicated dataset)?

Best,
BK
bhcho
 
Posts: 236
Joined: Mon Apr 05, 2010 2:15 pm

Re: image import python script

Postby jmoore » Mon Nov 21, 2011 7:37 pm

The use of "omero import" shouldn't re-create a session, but instead will re-use the same one as long as its active. If you'd prefer to use a python-script, however, you can do the following:

Code: Select all
import omero
import omero.cli

def method():
            cli = omero.cli.CLI()
            cli.loadplugins()
            cmd = ["-s", self.host, "-p", str(self.port), "-k", key, "import"]
            ## To re-direct output to a file
            ## cmd.extend([str("---errs=%s"%t), str("---file=%s"%to)])
            cmd.extend(shlex.split(self.importArgs))
            cmd.append(fileName)
            logging.debug("cli.invoke(%s)" % cmd)
            cli.invoke(cmd)
            retCode = cli.rv


This is from https://github.com/openmicroscopy/openmicroscopy/blob/master/components/tools/OmeroFS/fsDropBoxMonitorClient.py#L608

Cheers,
~Josh.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: image import python script

Postby bhcho » Fri Nov 25, 2011 4:50 pm

After several iterations of image importing and calculation (executed from my custom Bridge), OMERO.web looked slow so I checked Indexer log and I found lots of errors like below.
2011-11-23 17:57:30,526 INFO [ ome.services.fulltext.FullTextIndexer] (3-thread-4) INDEXED 13 objects in 1 batch(es) [38508 ms.]
2011-11-23 17:57:30,528 ERROR [ ome.services.util.ServiceHandler] (3-thread-4) Method interface ome.services.util.Executor$Work.doWork invocation took 38511
2011-11-23 17:57:33,075 INFO [ ome.services.fulltext.FullTextIndexer] (3-thread-1) INDEXED 1 objects in 1 batch(es) [1056 ms.]
2011-11-23 18:25:00,020 INFO [ome.services.sessions.state.SessionCache] (3-thread-3) Waiting for synchronization
2011-11-23 18:25:02,346 INFO [ome.services.sessions.state.SessionCache] (3-thread-5) Synchronizing session cache. Count = 2
2011-11-23 18:25:02,361 INFO [ome.services.sessions.state.SessionCache] (3-thread-5) Synchronization took 15 ms.
2011-11-23 18:55:04,018 INFO [ome.services.sessions.state.SessionCache] (3-thread-4) Waiting for synchronization
2011-11-23 18:55:05,350 INFO [ome.services.sessions.state.SessionCache] (3-thread-2) Synchronizing session cache. Count = 2
2011-11-23 18:55:05,367 INFO [ome.services.sessions.state.SessionCache] (3-thread-2) Synchronization took 17 ms.
2011-11-23 19:25:08,018 INFO [ome.services.sessions.state.SessionCache] (3-thread-1) Waiting for synchronization


from Blitz log,
2011-11-25 11:24:33,282 ERROR [ ome.services.blitz.impl.ServiceFactoryI] (l.Server-9) Error destroying servant: 80:2:cf:74:2845654e:133d13e045d:-1cd5omero.api.ThumbnailStore=om$
java.lang.RuntimeException: omero.InternalException
serverStackTrace = "java.util.NoSuchElementException
at java.util.LinkedList.remove(LinkedList.java:788)
at java.util.LinkedList.removeLast(LinkedList.java:144)
at ome.security.basic.CurrentDetails.logout(CurrentDetails.java:180)
at ome.security.basic.BasicSecurityWiring.logout(BasicSecurityWiring.java:110)
at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:85)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:40)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy69.close(Unknown Source)
at ome.services.blitz.impl.AbstractAmdServant.close_async(AbstractAmdServant.java:227)
at ome.services.blitz.impl.AbstractAmdServant.close(AbstractAmdServant.java:198)
at ome.services.blitz.impl.ServiceFactoryI.doDestroy(ServiceFactoryI.java:776)
at ome.services.blitz.impl.ServiceFactoryI.destroy(ServiceFactoryI.java:689)
at Glacier2._SessionDisp.___destroy(_SessionDisp.java:82)
at omero.api._ServiceFactoryDisp.__dispatch(_ServiceFactoryDisp.java:1231)
at IceInternal.Incoming.invoke(Incoming.java:159)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2037)
at Ice.ConnectionI.message(ConnectionI.java:972)
at IceInternal.ThreadPool.run(ThreadPool.java:577)
at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971)
"
serverExceptionClass = "java.util.NoSuchElementException"
message = (null)
at ome.services.blitz.impl.AbstractAmdServant$1.ice_exception(AbstractAmdServant.java:191)
at ome.services.blitz.impl.AbstractAmdServant.close_async(AbstractAmdServant.java:255)
at ome.services.blitz.impl.AbstractAmdServant.close(AbstractAmdServant.java:198)
at ome.services.blitz.impl.ServiceFactoryI.doDestroy(ServiceFactoryI.java:776)
at ome.services.blitz.impl.ServiceFactoryI.destroy(ServiceFactoryI.java:689)
at Glacier2._SessionDisp.___destroy(_SessionDisp.java:82)
at omero.api._ServiceFactoryDisp.__dispatch(_ServiceFactoryDisp.java:1231)
at IceInternal.Incoming.invoke(Incoming.java:159)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2037)
at Ice.ConnectionI.message(ConnectionI.java:972)
at IceInternal.ThreadPool.run(ThreadPool.java:577)
at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971)
Caused by: omero.InternalException
serverStackTrace = "java.util.NoSuchElementException
at java.util.LinkedList.remove(LinkedList.java:788)
at java.util.LinkedList.removeLast(LinkedList.java:144)
at ome.security.basic.CurrentDetails.logout(CurrentDetails.java:180)
at ome.security.basic.BasicSecurityWiring.logout(BasicSecurityWiring.java:110)
at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:85)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:40)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy69.close(Unknown Source)
at ome.services.blitz.impl.AbstractAmdServant.close_async(AbstractAmdServant.java:227)
at ome.services.blitz.impl.AbstractAmdServant.close(AbstractAmdServant.java:198)
at ome.services.blitz.impl.ServiceFactoryI.doDestroy(ServiceFactoryI.java:776)
at ome.services.blitz.impl.ServiceFactoryI.destroy(ServiceFactoryI.java:689)
at Glacier2._SessionDisp.___destroy(_SessionDisp.java:82)
at omero.api._ServiceFactoryDisp.__dispatch(_ServiceFactoryDisp.java:1231)
at IceInternal.Incoming.invoke(Incoming.java:159)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2037)
at Ice.ConnectionI.message(ConnectionI.java:972)
at IceInternal.ThreadPool.run(ThreadPool.java:577)
at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971)
"
serverExceptionClass = "java.util.NoSuchElementException"
message = (null)
at omero.util.IceMapper.handleException(IceMapper.java:1262)
... 12 more


Do you have any idea about this?

Best,
BK
bhcho
 
Posts: 236
Joined: Mon Apr 05, 2010 2:15 pm

Re: image import python script

Postby jmoore » Fri Nov 25, 2011 7:50 pm

It would look like something has happened to your indexer, though what you show here are just symptoms not the cause. Could you attach your entire log, and possibly the master.out and master.err files?

Thanks,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Next

Return to Developer Discussion

Who is online

Users browsing this forum: Google [Bot] and 0 guests