Querying the external XML file (written by OMERO.editor)

[bhcho]

Yes. I want to delete it after I retrieve the information. I don't want to get into trouble with the file permission.

So, is there anyway to retrieve the absolute path of "OMERO_TEMPDIR" in my java code?
because that path is public directory in my system.

BK

[wmoore]

The java file you write as described above will be written where you specify (as described), not in OMERO_TEMPDIR.

Write the file, read the file and delete.

// write file (as described)
path = file.getPath()
// print path

// read file
File f = new File(path)
f.read()

// delete
f.delete()

[bhcho]

I'm now trying to put my code to the Extensions.jar
But in my code, which extends BridgeHelper class, how can I get the omero.client.

my code is something like this

Code: Select all

public class ExperimentFileBridge extends BridgeHelper {
    @Override
    public void set(final String name, final Object value,
            final Document document, final LuceneOptions _opts) {
         if (value instanceof Image) {
            final Image i = (Image) value;
            Long iid = i.getId();

            // here I need to create ServiceFactoryPrx, IMetadataPrx, RawFileStorePrx
            // in order to use metadata.loadAnnotations API function.

           

In order to create ServiceFactoryPrx, I think I need to get a "omero.client" object from the user's login account. Could you tell me how to do it?

[jmoore]

Sorry, BK.

As I mentioned, there was some confusion in this thread because of all the topics covered. Code inside of the server (i.e. your bridge) should not use omero.client, since that's client side code. Can you post what you have currently got coded so I can make suggestions?

Cheers,
~Josh

[bhcho]

hi Josh,

Followed is what I have now.
I was trying to use this code (especially "FindExpAnnotation" function and corresponding "downloadFile" function)

Code: Select all

public class ParseExp {

    public static void main(String[] args) throws Exception {

        // Configuration for the client object in this case comes
        // from the ICE_CONFIG environment variable.
        omero.client client = new omero.client("xxx.xxx.xxx.xxx", 4064);
        client.createSession("xxxx", "xxxx");

        try {

            // Get or create a valid Image id.
            long imageId=0;
            if (args.length > 0) {
                imageId = Long.valueOf(args[0]);
            } else {
                 System.out.println("No image ID");
                 return;
            }

            FindExpAnnotation(client, imageId);

        } finally {
            client.closeSession();
        }
    }

    /**
     * Finds all Experiment Annotation file (XML) attached to the given image.
     */
    public static void FindExpAnnotation(omero.client client, long imageId) throws ServerError, Exception {

        final ServiceFactoryPrx factory = client.getSession();
        final IMetadataPrx metadata = factory.getMetadataService();
        final RawFileStorePrx rawfilestore = factory.createRawFileStore();
        String types = "ome.model.annotations.FileAnnotation";
        final Map<Long, List<IObject>> annotations = metadata.loadAnnotations(
                "Image", Collections.singletonList(imageId),
                Collections.singletonList(types), null, null);

        if (!annotations.containsKey(imageId)) {
            System.out.println("No annotations.");
            return;
        }
        String tagNs = null;
        String tagName = null;
        String subtagName = null;
       
        for (IObject obj : annotations.get(imageId)) {
            FileAnnotation ann = (FileAnnotation) obj;
            tagNs = ann.getNs().getValue();
            tagName = ann.getFile().getName().getValue();
            subtagName = tagName.substring(tagName.length()-7);
            if (tagNs.equalsIgnoreCase("openmicroscopy.org/omero/editor/experiment") && subtagName.equalsIgnoreCase("cpe.xml")){
                 OriginalFile originalfile = ann.getFile();
                 Long fileId = originalfile.getId().getValue();
                 Long fileSize = originalfile.getSize().getValue();
                 String filePath = originalfile.getName().getValue();
                 rawfilestore.setFileId(fileId);
                 
                 File file = new File("temp.xml");
                 downloadFile(rawfilestore, originalfile, file); // write the original file contents to the "temp.xml" file
                 
                 // from here, I need to parse the XML file and map it to Lucene Index
                 
                 file.delete();
            }
           
        }
    }
   
    public static void downloadFile(RawFileStorePrx rawfilestore, OriginalFile originalfile, File file) throws Exception {
         int INC = 262144;
         Long fileId = originalfile.getId().getValue();
         rawfilestore.setFileId(fileId);
         Long fileSize = originalfile.getSize().getValue();

                 
         String path = file.getAbsolutePath();
         int offset = 0;
         int length = fileSize.intValue();//Convert.ToInt32(fileSize);//(int)fileSize;
         try {
              FileOutputStream stream = new FileOutputStream(file);
              try {
                   try {
                        for (offset = 0; (offset+INC) < fileSize;) {
                             stream.write(rawfilestore.read(offset, INC));
                             offset += INC;
                        }
                   } finally {
                        stream.write(rawfilestore.read(offset, length-offset));
                        stream.close();
                   }
              } catch (Exception e) {
                   if (stream != null) stream.close();
                   if (file != null) file.delete();
              }
         } catch (IOException e) {
              if (file != null) file.delete();
              //closeService(rawfilestore); //store.close()
              rawfilestore.close();
              // throw new DSAccessException("Cannot create file " +path, e);
         }
         rawfilestore.close();//closeService(store); // store.close() throws an exception
    }
   
   
}

to the bridge

Code: Select all

public class ExperimentFileBridge extends BridgeHelper {
    @Override
    public void set(final String name, final Object value,
            final Document document, final LuceneOptions _opts) {
         if (value instanceof Image) {
            final Image i = (Image) value;
            Long iid = i.getId();

// from here, using the image ID, I need to find all Experiment XML file and parse and map to Lucene Index




[jmoore]

hi Josh,

Hey BK,

Followed is what I have now.
I was trying to use this code (especially "FindExpAnnotation" function and corresponding "downloadFile" function)

Unfortunately, this won't work. Or it might, but will be slow and error prone. A bridge is an server-internal OMERO component, and so will never want to access "omero.client" or any of the "*Prx" classes.

Code: Select all

public class ExperimentFileBridge extends BridgeHelper {
    @Override
    public void set(final String name, final Object value,
            final Document document, final LuceneOptions _opts) {
         if (value instanceof Image) {
            final Image i = (Image) value;
            Long iid = i.getId();

// from here, using the image ID, I need to find all Experiment XML file and parse and map to Lucene Index


Here you have an Image i which is attached to a Hibernate session, i.e. as you access fields it will read from the database. (As opposed to having an Image on the client side, where what you have is all you're getting).

First you need to get the annotations and look for a FileAnnotation:

Code: Select all

for (Annotation a : i.linkedAnnotationList()) {
   ...
}


That is identical to the code in FullTextBridge.java.

Once you have the FileAnnotation you need to get the original file:

Code: Select all

ome.model.core.OriginalFile file = fa.getFile();


With that you can use other internal OMERO services to get the path to the file itself. Examples of this can be seen in BridgeHelper:

• addContent which calls
• parse(), which uses an instance of OriginalFilesService

There are cleaner ways for you to get access to the OriginalFileService via configuration, but for the moment, here's a shortcut:

Code: Select all

ome.io.nio.OriginalFilesService files = (ome.io.nio.OriginalFilesService) ome.system.OmeroContext.getManagedServerContext().getBean("/OMERO/Files");


Hope that helps.
~J.

[jmoore]

Thanks Josh,
I'll try with your answer.
But the problem is that it's very hard for me to find the appropriate page
in javadoc.

For example, I want to look at all methods from
ome.model.annotations.Annotation class.
But I don't know where to see from
http://hudson.openmicroscopy.org.uk/job/OMERO/javadoc/

The generated model objects are not in the javadocs, BK, since they all follow a common scheme, i.e. they are generated from a central definition:

http://git.openmicroscopy.org/?p=ome.git;a=tree;f=components/model/resources/mappings

The ObjectModel page describes more about this. Important to realize is that Annotation is an abstract-class, You'll need to look at the concrete type FileAnnotation to find the getFile() method.

Cheers,
~Josh.

[bhcho]

hi Josh,

for the OriginalFilesService,

Code: Select all

ome.io.nio.OriginalFilesService files = (ome.io.nio.OriginalFilesService) ome.system.OmeroContext.getManagedServerContext().getBean("/OMERO/Files");

1.1.
I don't understand the code above.
could you explain how to create an object for OriginalFilesService again?
(the first line is not able to be compiled anyway)

1.2.
what does the second line do?


for the parse function

Code: Select all

237     protected Iterable<Reader> parse(final OriginalFile file,
238             final OriginalFilesService files,
239             final Map<String, FileParser> parsers) {
240         if (files != null && parsers != null) {
241             if (file != null && file.getMimetype() != null) {
242                 String path = files.getFilesPath(file.getId());
243                 String format = file.getMimetype();
244                 FileParser parser = parsers.get(format);
245                 if (parser != null) {
246                     return parser.parse(new File(path));
247                 } else {
248                     parser = parsers.get("*");
249                     if (parser != null) {
250                         return parser.parse(new File(path));
251                     }
252                 }
253             }
254         }
255         return FileParser.EMPTY;
256     }

2.1.
what is the FileParser class? How can I import this class?
I was trying to understand it from http://trac.openmicroscopy.org.uk/ome/wiki/FileParsers, but those links in the webpage are all broken.

2.2.
what is the return output of the FileParser.parse() function? is it the loaded XML file? I want to know exactly what type of the return output is, because I need to parse the XML file later. Hopefully it could be a File object. (before you suggested this, I created an empty File object and loaded the content of the original file using RawFileStorePrx and FileOutputStream like the following code, which turned out to be wrong in this purpose)

Code: Select all

    public static void downloadFile(RawFileStorePrx rawfilestore, OriginalFile originalfile, File file) throws Exception {
         int INC = 262144;
         Long fileId = originalfile.getId().getValue();
         rawfilestore.setFileId(fileId);
         Long fileSize = originalfile.getSize().getValue();

                 
         String path = file.getAbsolutePath();
         int offset = 0;
         int length = fileSize.intValue();//Convert.ToInt32(fileSize);//(int)fileSize;
         try {
              FileOutputStream stream = new FileOutputStream(file);
              try {
                   try {
                        for (offset = 0; (offset+INC) < fileSize;) {
                             stream.write(rawfilestore.read(offset, INC));
                             offset += INC;
                        }
                   } finally {
                        stream.write(rawfilestore.read(offset, length-offset));
                        stream.close();
                   }
              } catch (Exception e) {
                   if (stream != null) stream.close();
                   if (file != null) file.delete();
              }
         } catch (IOException e) {
              if (file != null) file.delete();
              //closeService(rawfilestore); //store.close()
              rawfilestore.close();
              // throw new DSAccessException("Cannot create file " +path, e);
         }
         rawfilestore.close();//closeService(store); // store.close() throws an exception
    }

3.
Since this code is the server side code, I'm not sure how I can easily debug this. Do you have any method that I can debug the code like launching the script from server? (otherwise, I dont have any clue or error message when it does not work)

[jmoore]

hi Josh,

for the OriginalFilesService,

Code: Select all

ome.io.nio.OriginalFilesService files = (ome.io.nio.OriginalFilesService) ome.system.OmeroContext.getManagedServerContext().getBean("/OMERO/Files");

1.1.
I don't understand the code above.
could you explain how to create an object for OriginalFilesService again?
(the first line is not able to be compiled anyway)

What error are you getting?

1.2.
what does the second line do?

The two lines belong together. They acquire an object of type OriginalFilesService.

for the parse function

Code: Select all

237     protected Iterable<Reader> parse(final OriginalFile file,
238             final OriginalFilesService files,
239             final Map<String, FileParser> parsers) {
240         if (files != null && parsers != null) {
241             if (file != null && file.getMimetype() != null) {
242                 String path = files.getFilesPath(file.getId());
243                 String format = file.getMimetype();
244                 FileParser parser = parsers.get(format);
245                 if (parser != null) {
246                     return parser.parse(new File(path));
247                 } else {
248                     parser = parsers.get("*");
249                     if (parser != null) {
250                         return parser.parse(new File(path));
251                     }
252                 }
253             }
254         }
255         return FileParser.EMPTY;
256     }

2.1.
what is the FileParser class? How can I import this class?
I was trying to understand it from http://trac.openmicroscopy.org.uk/ome/wiki/FileParsers, but those links in the webpage are all broken.

FileParser is a way to turn a file into a single stream of characters that Lucene can understand. This doesn't completely fulfill your requirement, since you need to turn a file into key/value pairs. But where you see FileParser being used, you can imagine that that's the code you have written or will be writing.

By the way, thanks for pointing out the broken links; that came from our recent move to git. The paths to the files, however, would have been the same in Eclipse or similar.

2.2.
what is the return output of the FileParser.parse() function? is it the loaded XML file? I want to know exactly what type of the return output is, because I need to parse the XML file later. Hopefully it could be a File object. (before you suggested this, I created an empty File object and loaded the content of the original file using RawFileStorePrx and FileOutputStream like the following code, which turned out to be wrong in this purpose)

parse() returns an Iterable of Readers which can be turned into a list of strings. Again, this won't fully suffice in your case.

3.
Since this code is the server side code, I'm not sure how I can easily debug this. Do you have any method that I can debug the code like launching the script from server? (otherwise, I dont have any clue or error message when it does not work)

You can try by running SearchTest.java via testng while in Eclipse or copying it for your own use.

Cheers,
~Josh

[bhcho]

Thanks Josh,

the two lines belong together. They acquire an object of type OriginalFilesService.

This is my bad. Sorry.

And I finally compiled my code, generated a jar file and copied it to the server.

after I tried to linking an Experiment file to an image, I tried to query from OMERO shell like

Code: Select all

import omero

host = "xxx"
port = 4064
user = "yyy"
password = "zzz"

client = omero.client(host, port)
session = client.createSession(user, password)

search = client.sf.createSearchService()
search.onlyType("Image")
search.byFullText("Slide.Protocol.Substrate:G*")

ids = [x.id.val for x in search.results()]


but I got error.

when I look at the Indexer-0.log, I see the following error message,

2011-02-14 14:09:44,260 ERROR [ ome.services.fulltext.FullTextBridge] (3-thread-5) Error calling set on custom bridge type:class edu.cmu.search.bridges.ExperimentFileBridge; instance:edu.cmu.search.bridges.ExperimentFileBridge@7dc68ff4
java.lang.NullPointerException
at edu.cmu.search.bridges.ExperimentFileBridge.set(ExperimentFileBridge.java:104)
at ome.services.fulltext.FullTextBridge.set_custom(FullTextBridge.java:314)
at ome.services.fulltext.FullTextBridge.set(FullTextBridge.java:126)
at ome.util.DetailsFieldBridge.set(DetailsFieldBridge.java:49)
at org.hibernate.search.engine.DocumentBuilderIndexedEntity.buildDocumentFields(DocumentBuilderIndexedEntity.java:396)
at org.hibernate.search.engine.DocumentBuilderIndexedEntity.getDocument(DocumentBuilderIndexedEntity.java:380)
at org.hibernate.search.engine.DocumentBuilderIndexedEntity.createAddWork(DocumentBuilderIndexedEntity.java:328)
at org.hibernate.search.engine.DocumentBuilderIndexedEntity.addWorkToQueue(DocumentBuilderIndexedEntity.java:317)
at org.hibernate.search.backend.impl.BatchedQueueingProcessor.addWorkToBuilderQueue(BatchedQueueingProcessor.java:153)
at org.hibernate.search.backend.impl.BatchedQueueingProcessor.processWorkByLayer(BatchedQueueingProcessor.java:140)
at org.hibernate.search.backend.impl.BatchedQueueingProcessor.prepareWorks(BatchedQueueingProcessor.java:128)
at org.hibernate.search.backend.impl.BatchedQueueingProcessor.add(BatchedQueueingProcessor.java:111)
at org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization.add(PostTransactionWorkQueueSynchronization.java:39)
at org.hibernate.search.backend.impl.TransactionalWorker.performWork(TransactionalWorker.java:50)
at org.hibernate.search.impl.FullTextSessionImpl.index(FullTextSessionImpl.java:153)
at ome.services.fulltext.FullTextIndexer$Index.go(FullTextIndexer.java:82)
at ome.services.fulltext.FullTextIndexer.doIndexing(FullTextIndexer.java:202)
at ome.services.fulltext.FullTextIndexer.doIndexingWithWorldRead(FullTextIndexer.java:162)
at ome.services.fulltext.FullTextIndexer.doWork(FullTextIndexer.java:143)
at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:409)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.security.basic.NullEventHandler.invoke(NullEventHandler.java:39)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:231)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:111)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy56.doWork(Unknown Source)
at ome.services.util.Executor$Impl.execute(Executor.java:339)
at ome.services.fulltext.FullTextThread.doRun(FullTextThread.java:135)
at ome.services.util.ExecutionThread.run(ExecutionThread.java:58)
at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)
at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:264)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

the line 104 of "ExperimentFileBridge.java" is

Code: Select all

map =  exParser.parse(new File(path));

exParser is a my new class that mimics the "FileParser.java".
the problem is that I dont know how to follow the error from here.
I need to trace the code and find what part of my code is wrong.

Best,
BK