Querying the external XML file (written by OMERO.editor)

[bhcho]

thanks.

The fields you'll want to examine on the OriginalFile include: name and mimetype. Where name will (probably) be of the form "*.cpe.xml" and mimetype will be "text/xml". You might also check the value of "FileAnnotation.ns" depending on how the protocol files were attached.

1.
Because of my stupidness and laziness, it is so hard to find appropriate sample codes sometimes.
Before coding it in Java, I was trying to test it from omero shell in python.

assuming the Experiment xml file is attached to image 4519,

Code: Select all

idString = str(4519)
queryService = session.getQueryService()
query_string = "select i from Image i where i.id in (%s)" % idString
images = queryService.findAllByQuery(query_string, None)
img=images[0]

metadata = session.getMetadataService()
annotations = metadata.loadAnnotations("Image",str(4519),"ome.model.annotations.FileAnnotation","null","null")

ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (26, 0))

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

/usr0/local/omero.server/OMERO.server-Beta-4.2.1/lib/scripts/<ipython console> in <module>()

/usr0/local/omero.server/OMERO.server-Beta-4.2.1/lib/python/omero_api_IMetadata_ice.pyc in loadAnnotations(self, rootType, rootIds, annotationTypes, annotatorIds, options, _ctx)
     92
     93         def loadAnnotations(self, rootType, rootIds, annotationTypes, annotatorIds, options, _ctx=None):
---> 94             return _M_omero.api.IMetadata._op_loadAnnotations.invoke(self, ((rootType, rootIds, annotationTypes, annotatorIds, options), _ctx))
     95
     96         def loadAnnotations_async(self, _cb, rootType, rootIds, annotationTypes, annotatorIds, options, _ctx=None):

ValueError: expected sequence value


I don't know how to use the loadAnnotations API function in python. I think I'm wrong with the input arguments.
And after getting the annotations, can you please let me know how to examine the file name and the mimetype? (I guess I can use the annotaion.getName().getValue() for the file name)

2.
Could you let me know the java version of it?


I'm so sorry to keep bothering you.

Sincerely,
BK

[wmoore]

Hi,

Here's come python code from the Thumbnail_Figure.py script.

Code: Select all

imageIds = []
imageIds.append( image.getId().getValue() )
types = ["ome.model.annotations.TagAnnotation"]
annotations = metadataService.loadAnnotations("Image", ImageIds, types, None, None)

for imageId, tags in annotations.items():
            for tag in tags:
                tagId = tag.getId().getValue()


Hope that helps,

Will.

[bhcho]

Thanks Will and Josh,

I figured out.

Code: Select all

idString = str(4519)
queryService = session.getQueryService()
query_string = "select i from Image i where i.id in (%s)" % idString
images = queryService.findAllByQuery(query_string, None)
img=images[0]


metadata = session.getMetadataService()
imageIds = []
imageIds.append( img.getId().getValue() )
types = ["ome.model.annotations.FileAnnotation"]
annotations = metadata.loadAnnotations("Image", imageIds, types, None, None)

for imageId, tags in annotations.items():
            for tag in tags:
                  tagNs = tag.getNs().getValue()
                  tagName = tag.getFile().getName().getValue()
                  if (tagNs == "openmicroscopy.org/omero/editor/experiment") and (tagName[-7:].lower() == "cpe.xml"):
                   tagId = tag.getId().getValue()
                   tagId
                   tag.getDescription().getValue()

by doing "tag.getDescription().getValue()",
I got

'<protocol>\r\n <n>Sample</n>\r\n <d>This element describes the components with which the biological sample is prepared before the acquisition.\n</d>\r\n <ss>\r\n <s l="1" n="Protocol">\r\n <p>\r\n <n>Reference</n>\r\n <v/>\r\n </p>\r\n <p>\r\n <n>Fixation</n>\r\n <v>none</v>\r\n </p>\r\n <p>\r\n <n>Permeabilization</n>\r\n <v>none</v>\r\n </p>\r\n <p>\r\n <n>Substrate</n>\r\n <v>Glass</v>\r\n </p>\r\n <p>\r\n <n>Temperature (&#xba;C)</n>\r\n <v>20</v>\r\n </p>\r\n <p>\r\n <n>Description</n>\r\n <v>Media was aspirated and replaced with Optimem containing Hoechst (at 0.5 microliter/mL concentration) followed by 45 minutes incubation before imaging.</v>\r\n </p>\r\n <p>\r\n <n>Author</n>\r\n <v/>\r\n </p>\r\n <p>\r\n <n>Title</n>\r\n <v>3T3 RandTag Protocol 7</v>\r\n </p>\r\n </s>\r\n <s l="1" n="Cell_Type">\r\n <p>\r\n <n>Name</n>\r\n <v>3T3-CDtag-CZ1E5</v>\r\n </p>\r\n <p>\r\n <n>Organism</n>\r\n <v>Mus musculus</v>\r\n </p>\r\n <p>\r\n <n>atcc_number</n>\r\n <v>CRL-1658</v>\r\n </p>\r\n <p>\r\n <n>cell_mesh_heading</n>\r\n <v>NIH 3T3 Cells</v>\r\n </p>\r\n <p>\r\n <n>cell_mesh_tree_number</n>\r\n <v>A11.251.210.700.550</v>\r\n </p>\r\n </s>\r\n </ss>\r\n</protocol>\r\n'

which is what I saved in the Experiment xml file. I was so happy that OMERO automatically parse the xml file and save it in this way.
Now I only need to parse this xml-like text and map all the values into my new Lucene Index.

thanks again.

BK

[wmoore]

Hi BK,

I'm sorry to disappoint you, but the description you got there is just a 'preview' of the OMERO.editor file, in order to provide the "preview" functionality you see in Insight when you attach an Editor file to a Project, Dataset or Image.

The description is limited to a certain length (I forget what it is now, but you can easily see with a larger Editor file that it will be truncated). It's even using different XML tags from a proper Editor file. <p> is parameter, <n> is name and <v> is value, just to be more concise. It's also possible that these tags etc will change without notice!

Also, it's not generated if (for example) you save an Editor file locally, then use Insight to upload and attach it. Having said that - if those conditions don't worry you then you are OK!

If you need the full file, you will have to do

Code: Select all

for imageId, anns in annotations.items():
    for a in anns:
        a.getFile()    # returns the Original File.           


Then you need to download the file and you can then read it!
I'm not sure if the file returned above is loaded, so you might need to load the file etc - as described in another of your posts https://www.openmicroscopy.org.uk/commu ... 5a050ba61d

One more step closer!

Will.

[bhcho]

what is the equivalent java code for the following?

Code: Select all

filePath = scriptUtil.downloadFile(rawFileStore, originalFile)

I dont know what package should I import for "scriptUtil"
in python it was

Code: Select all

import omero.util.script_utils as scriptUtil

and what is the type of the returned "filepath"

[wmoore]

Hi

There is no equivalent to the Python script_utils in Java. You will have to look at the code in script_utils https://trac.openmicroscopy.org.uk/omer ... t_utils.py and see if you can do something similar in Java.

The API behaves a similar way in Python and Java. E.g. originalFile.getSize().getValue() and rawFileStore.read(count, blockSize) should give you equivalent output in both languages.

Code: Select all


    fileId = originalFile.getId().getValue()
    rawFileStore.setFileId(fileId)
    fileSize = originalFile.getSize().getValue()
    maxBlockSize = 10000
    cnt = 0

    fileHandle = open(filePath, 'w')
    data = '';
    cnt = 0;
    fileSize = originalFile.getSize().getValue()
    while(cnt<fileSize):
        blockSize = min(maxBlockSize, fileSize)
        block = rawFileStore.read(cnt, blockSize)
        cnt = cnt+blockSize
        fileHandle.write(block)
    fileHandle.close()
    return filePath



RawFileStore API is here
http://hudson.openmicroscopy.org.uk/job ... wFileStore

Good luck!

Will.

[bhcho]

it seems like
in order to retrieve the file contents, I have to create another temporary file and copy the file contents to the temporary file first. Then I load the temporary file. Am I right?

But in java, the rawfilestore.read(a,b) function does not work.
here's my code and the compile error

Code: Select all

    public static void FindExpAnnotation(omero.client client, long imageId) throws ServerError, Exception {

        final ServiceFactoryPrx factory = client.getSession();
        final IMetadataPrx metadata = factory.getMetadataService();
        final RawFileStorePrx rawfilestore = factory.createRawFileStore();
        String types = "ome.model.annotations.FileAnnotation";
        final Map<Long, List<IObject>> annotations = metadata.loadAnnotations(
                "Image", Collections.singletonList(imageId),
                Collections.singletonList(types), null, null);

        if (!annotations.containsKey(imageId)) {
            System.out.println("No annotations.");
            return;
        }
        String tagNs = null;
        String tagName = null;
        String subtagName = null;
        for (IObject obj : annotations.get(imageId)) {
            FileAnnotation ann = (FileAnnotation) obj;
            tagNs = ann.getNs().getValue();
            tagName = ann.getFile().getName().getValue();
            subtagName = tagName.substring(tagName.length()-7);
            if (tagNs.equalsIgnoreCase("openmicroscopy.org/omero/editor/experiment") && subtagName.equalsIgnoreCase("cpe.xml")){
                 OriginalFile originalfile = ann.getFile();
                 Long fileId = originalfile.getId().getValue();
                 Long fileSize = originalfile.getSize().getValue();
                 String filePath = originalfile.getName().getValue();
                 rawfilestore.setFileId(fileId);
                 byte[] block = rawfilestore.read(0,fileSize);
                 
                 // write block to a temporary file

//                 BufferedOutputStream bytesOut = new BufferedOutputStream(new FileOutputStream("bytes.dat"));
//                 
//                 File file = new File(filePath);
//                 DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
//                 DocumentBuilder db = dbf.newDocumentBuilder();
//                 Document doc = db.parse(file);
//                 doc.getDocumentElement().normalize();
//                 System.out.println("Root element " + doc.getDocumentElement().getNodeName());
            }
           
        }
    }

File: C:\ParseExp.java [line: 106]
Error: C:\ParseExp.java:106: cannot find symbol
symbol : method read(int,java.lang.Long)
location: interface omero.api.RawFileStorePrx

the line 106 is

Code: Select all

byte[] block = rawfilestore.read(0,fileSize);

[jburel]

Hi

To load the content of the file, follow some pseudo-code I use.

int INC = 262144;
//Create a raw file store
RawFileStorePrx store;
store.setFileId(fileID); //the id of the original file.
File file = new File("MyFolder/test.txt"); // The file where to write the data returned by the server
String path = file.getAbsolutePath();
int offset = 0;
int length = (int) size; //where size is the size of the file stored.
try {
FileOutputStream stream = new FileOutputStream(file);
try {
try {
for (offset = 0; (offset+INC) < size;) {
stream.write(store.read(offset, INC));
offset += INC;
}
} finally {
stream.write(store.read(offset, length-offset));
stream.close();
}
} catch (Exception e) {
if (stream != null) stream.close();
if (file != null) file.delete();
}
} catch (IOException e) {
if (file != null) file.delete();
closeService(store); //store.close()
throw new DSAccessException("Cannot create file " +path, e);
}
closeService(store); // store.close() throws an exception

You will then have the content of the file stored server side
copied in your local file.

Jmarie

[bhcho]

Hi BK,

1.
where is this 'test.txt' file created in the server (assuming this code is automatically called as "extensions.jar" in the server)?
would it be created where the 'extensions.jar' file is? or somewhere else?

to clarify, are you still talking about how to get the file from within your bridge? If so, much of this thread should not be used. When writing a bridge, you are internal to the server and should definitely NOT be using any of the *Prx interfaces. Further, files will be created relative to OMERO_HOME, where you probably don't want to be writing. Use omero.util.TempFileManager to create a path that you can write to safely and which will be cleaned up if an exception is thrown (though you should close and delete it yourself).

By the way, TempFileManager has a method "getTempDir()" but you will need to create your own via: new TempFileManager("cmu-bridge").

Cheers,
~Josh

[wmoore]

I think the file will be created where the Java client code is running.
If you want to specify a different location on the client machine, use an absolute file path:
E.g. "/Users/name/Desktop/downloads/temp/file.xml"

You will probably want to read it straight away and delete it?

Will