We're Hiring!

Querying the external XML file (written by OMERO.editor)

General and open developer discussion about using OMERO APIs from C++, Java, Python, Matlab and more! Please new questions at https://forum.image.sc/tags/omero
Please note:
Historical discussions about OMERO. Please look for and ask new questions at https://forum.image.sc/tags/omero

If you are having trouble with custom code, please provide a link to a public repository, ideally GitHub.

Re: Querying the external XML file (written by OMERO.editor)

Postby bhcho » Fri Jan 28, 2011 4:47 pm

jmoore wrote:At this point, I would assume that you are in fact adding things to the Lucene index, but the question is whether or not what you are adding is what you are searching for. You might try http://www.getopt.org/luke/ to examine the /OMERO/FullText directory. Find an image that has passed through your bridge and look at its contents. It should have a field of the for A.B.C (or similar).


Unfortunately, I dont see a field of "image_name" for that image (which I tried to add) in the luke.
does this mean my code has some problem?

my code is exactly the same with the sample code
Code: Select all
/*
*   $Id$
*
*   Copyright 2008 Glencoe Software, Inc. All rights reserved.
*   Use is subject to license terms supplied in LICENSE.txt
*/

package edu.cmu.search.bridges;

import java.util.ArrayList;
import java.util.List;

import ome.model.containers.Dataset;
import ome.model.containers.DatasetImageLink;
import ome.model.containers.Project;
import ome.model.containers.ProjectDatasetLink;
import ome.model.core.Image;
import ome.services.fulltext.BridgeHelper;
import ome.services.fulltext.SimpleLuceneOptions;

import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.hibernate.search.bridge.FieldBridge;
import org.hibernate.search.bridge.LuceneOptions;

/**
* Example custom {@link FieldBridge} implementation which parses all
* {@link Image} names from a {@link Project} and inserts them into the index
* for that {@link Project}.
*
* @author Josh Moore, josh at glencoesoftware.com
* @since 3.0-Beta3
*/
public class ExperimentFileBridge extends BridgeHelper {

    /**
     * If the "value" argument is a {@link Project}, this
     * {@link FieldBridge bridge} gathers all images and adds them to the index
     * with a slightly reduced boost value. The field name of the image name is
     * "image_name" but the values are also added to the
     * {@link BridgeHelper#COMBINED} field via the
     * {@link #add(Document, String, String, org.apache.lucene.document.Field.Store, org.apache.lucene.document.Field.Index, Float)}
     * method.
     */
    @Override
    public void set(final String name, final Object value,
            final Document document, final LuceneOptions _opts) {

        if (value instanceof Project) {

            logger().info("Indexing all image names for " + value);

            // Copying lucene options with a new boost value
            final float reduced_boost = _opts.getBoost().floatValue() / 2;
            final LuceneOptions opts = new SimpleLuceneOptions(_opts, reduced_boost);

            final Project p = (Project) value;
            for (final ProjectDatasetLink pdl : p.unmodifiableDatasetLinks()) {
                final Dataset d = pdl.child();
                for (final DatasetImageLink dil : d.unmodifiableImageLinks()) {
                    final Image i = dil.child();

                    // Name is never null, but as an example it is important
                    // to always check the value for null, and either simply
                    // not call add() or to use a null token like "null".
                    if (i.getName() != null) {
                        add(document, "image_name", i.getName(), opts);
                    } else {
                        add(document, "image_name", "null", opts);
                    }
                }
            }
        } else if (value instanceof Image) {

            logger().info(
                    "Scheduling all project containers of " + value
                            + " for re-indexing");

            final Image i = (Image) value;
            final List<Project> list = new ArrayList<Project>();

            for (final DatasetImageLink dil : i.unmodifiableDatasetLinks()) {
                final Dataset d = dil.parent();
                for (final ProjectDatasetLink pdl : d
                        .unmodifiableProjectLinks()) {
                    list.add(pdl.parent());
                }
            }
            if (list.size() > 0) {
                // ticket:955 Disabling for the moment.
                // reindexAll(list);
            }
        }
    }
}
bhcho
 
Posts: 236
Joined: Mon Apr 05, 2010 2:15 pm

Re: Querying the external XML file (written by OMERO.editor)

Postby jmoore » Mon Jan 31, 2011 11:04 am

Hi BK,

the field "image_name" should be on any Project which is linked to that Image, not on the Image itself.

I took the following steps:
  • bin/omero config set omero.search.bridges ome.services.fulltext.bridges.ProjectWithImageNameBridge
  • Restarted the server
  • Ran the script below as root.

Code: Select all
rs = omero.rtypes.rstring
p = omero.model.ProjectI()
d = omero.model.DatasetI()
i = omero.model.ImageI()
p.name = rs("bkp")
p.linkDataset(d)
d.name = rs("bkd")
d.linkImage(i)
i.name = rs("bki")
i.acquisitionDate = omero.rtypes.rtime(0)
up = client.sf.getUpdateService()
p = up.saveAndReturnObject(p)
up.indexObject(p)

x = client.sf.createSearchService()
x.onlyType("Project")
x.byFullText("image_name:bk*")
print x.next()


and saw the project printed out. If you want to just search the "name" field of Image, you can use "name:bk" with onlyType("Image").

Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Querying the external XML file (written by OMERO.editor)

Postby bhcho » Mon Jan 31, 2011 3:36 pm

I have the following error when I ran the script as root
In [10]: rs = omero.rtypes.rstring

In [11]: p = omero.model.ProjectI()

In [12]: d = omero.model.DatasetI()

In [13]: i = omero.model.ImageI()

In [14]: p.name = rs("bkp")

In [15]: p.linkDataset(d)
Out[15]:
object #0 (::omero::model::ProjectDatasetLink)
{
_id = <nil>
_details = object #1 (::omero::model::Details)
{
_owner = <nil>
_group = <nil>
_creationEvent = <nil>
_updateEvent = <nil>
_permissions = <nil>
_externalInfo = <nil>
}
_loaded = True
_version = <nil>
_parent = object #2 (::omero::model::Project)
{
_id = <nil>
_details = object #3 (::omero::model::Details)
{
_owner = <nil>
_group = <nil>
_creationEvent = <nil>
_updateEvent = <nil>
_permissions = <nil>
_externalInfo = <nil>
}
_loaded = True
_version = <nil>
_datasetLinksSeq =
{
[0] = <object #0>
}
_datasetLinksLoaded = True
_datasetLinksCountPerOwner = {}
_annotationLinksSeq =
{
}
_annotationLinksLoaded = True
_annotationLinksCountPerOwner = {}
_name = object #4 (::omero::RString)
{
_val = bkp
}
_description = <nil>
}
_child = object #5 (::omero::model::Dataset)
{
_id = <nil>
_details = object #6 (::omero::model::Details)
{
_owner = <nil>
_group = <nil>
_creationEvent = <nil>
_updateEvent = <nil>
_permissions = <nil>
_externalInfo = <nil>
}
_loaded = True
_version = <nil>
_projectLinksSeq =
{
[0] = <object #0>
}
_projectLinksLoaded = True
_projectLinksCountPerOwner = {}
_imageLinksSeq =
{
}
_imageLinksLoaded = True
_imageLinksCountPerOwner = {}
_annotationLinksSeq =
{
}
_annotationLinksLoaded = True
_annotationLinksCountPerOwner = {}
_name = <nil>
_description = <nil>
}
}

bhcho
 
Posts: 236
Joined: Mon Apr 05, 2010 2:15 pm

Re: Querying the external XML file (written by OMERO.editor)

Postby jmoore » Mon Jan 31, 2011 3:48 pm

Hi BK,

Is there any more output? That's not an error, but just the printing of the object structure.

Cheers,
Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Querying the external XML file (written by OMERO.editor)

Postby bhcho » Mon Jan 31, 2011 4:03 pm

the following outputs are
In [20]: up = client.sf.getUpdateService()

In [21]: p = up.saveAndReturnObject(p)

In [22]: up.indexObject(p)

In [23]: x = client.sf.createSearchService()

In [24]: x.onlyType("Project")

In [25]: x.byFullText("image_name:bk*")

In [26]: print x.next()
object #0 (::omero::model::Project)
{
_id = object #1 (::omero::RLong)
{
_val = 551
}
_details = object #2 (::omero::model::Details)
{
_owner = object #3 (::omero::model::Experimenter)
{
_id = object #4 (::omero::RLong)
{
_val = 2
}
_details = <nil>
_loaded = False
_version = <nil>
_groupExperimenterMapSeq =
{
}
_groupExperimenterMapLoaded = False
_omeName = <nil>
_firstName = <nil>
_middleName = <nil>
_lastName = <nil>
_institution = <nil>
_email = <nil>
_annotationLinksSeq =
{
}
_annotationLinksLoaded = False
_annotationLinksCountPerOwner =
{
}
}
_group = object #5 (::omero::model::ExperimenterGroup)
{
_id = object #6 (::omero::RLong)
{
_val = 0
}
_details = <nil>
_loaded = False
_version = <nil>
_name = <nil>
_groupExperimenterMapSeq =
{
}
_groupExperimenterMapLoaded = False
_annotationLinksSeq =
{
}
_annotationLinksLoaded = False
_annotationLinksCountPerOwner =
{
}
_description = <nil>
}
_creationEvent = object #7 (::omero::model::Event)
{
_id = object #8 (::omero::RLong)
{
_val = 43210
}
_details = object #9 (::omero::model::Details)
{
_owner = <nil>
_group = <nil>
_creationEvent = <nil>
_updateEvent = <nil>
_permissions = object #10 (::omero::model::Permissions)
{
_perm1 = -547
}
_externalInfo = <nil>
}
_loaded = True
_status = <nil>
_time = object #11 (::omero::RTime)
{
_val = 1296487313669
}
_experimenter = <object #3>
_experimenterGroup = <object #5>
_type = object #12 (::omero::model::EventType)
{
_id = object #13 (::omero::RLong)
{
_val = 4
}
_details = <nil>
_loaded = False
_value = <nil>
}
_containingEvent = <nil>
_logsSeq =
{
}
_logsLoaded = False
_session = object #14 (::omero::model::Session)
{
_id = object #15 (::omero::RLong)
{
_val = 5403
}
_details = <nil>
_loaded = False
_version = <nil>
_node = <nil>
_uuid = <nil>
_owner = <nil>
_timeToIdle = <nil>
_timeToLive = <nil>
_started = <nil>
_closed = <nil>
_message = <nil>
_defaultEventType = <nil>
_userAgent = <nil>
_eventsSeq =
{
}
_eventsLoaded = False
_annotationLinksSeq =
{
}
_annotationLinksLoaded = False
_annotationLinksCountPerOwner =
{
}
}
}
_updateEvent = <object #7>
_permissions = object #16 (::omero::model::Permissions)
{
_perm1 = -103
}
_externalInfo = <nil>
}
_loaded = True
_version = <nil>
_datasetLinksSeq =
{
}
_datasetLinksLoaded = False
_datasetLinksCountPerOwner =
{
}
_annotationLinksSeq =
{
}
_annotationLinksLoaded = False
_annotationLinksCountPerOwner =
{
}
_name = object #17 (::omero::RString)
{
_val = bkp
}
_description = <nil>
}
bhcho
 
Posts: 236
Joined: Mon Apr 05, 2010 2:15 pm

Re: Querying the external XML file (written by OMERO.editor)

Postby jmoore » Mon Jan 31, 2011 4:10 pm

That looks like a success. The call to x.next() returned the project the script was looking for.

~J.
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Querying the external XML file (written by OMERO.editor)

Postby bhcho » Mon Jan 31, 2011 4:22 pm

oh...you are right.
I got the project IDs with the followings.
In [14]: ids = [y.id.val for y in x.results()]

In [15]: ids
Out[15]: [551L, 552L, 553L]


Thanks so much Josh!!

and I got the following results too.
In [40]: x = client.sf.createSearchService()

In [41]: x.onlyType("Image")

In [42]: x.byFullText("image_name:bk*")

In [43]: x.hasNext()
Out[43]: False

In [44]: x = client.sf.createSearchService()

In [45]: x.onlyType("Image")

In [46]: x.byFullText("name:bk*")

In [47]: ids = [y.id.val for y in x.results()]

In [48]: ids
Out[48]: [1L, 52L, 3801L, 4901L, 4951L, 4952L, 4953L]


does this mean that the image object will be indexed (automatically by some other internal Lucece mapper ?) by "name" field?
bhcho
 
Posts: 236
Joined: Mon Apr 05, 2010 2:15 pm

Re: Querying the external XML file (written by OMERO.editor)

Postby jmoore » Mon Jan 31, 2011 7:50 pm

bhcho wrote:does this mean that the image object will be indexed (automatically by some other internal Lucece mapper ?) by "name" field?


Right. The default bridge (FullTextBridge) always runs in addition to anything you may configure via bin/omero config. It parses fields as outlined under OmeroSearch.

Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

Re: Querying the external XML file (written by OMERO.editor)

Postby bhcho » Mon Jan 31, 2011 9:27 pm

Thanks so much again.

Another questions.

1.
Can I get the value of a specific field for a specific image object?
for example, can I retrieve the value of "A.B.C1" field for an Image object of 5309? (rather than retrieve all the image IDs that has the value of "Bob" for "A.B.C1" field)

2.
As you know, I want to parse the Experiment (XML) file and map all the values to the Lucene Index.
I think I need to use the following part to recognize if the OriginalFile is the Experiment file or not (rather than another generic XML file).
Code: Select all
    public void set(final String name, final Object value,
            final Document document, final LuceneOptions _opts) {

        if (value instanceof OriginalFile) {
             // 1. to recognize if it's the experiment file. If it's not, do nothing from here.
             // 2. to parse the XML file
             // 3. map all the parsed values to Lucene Index
         }


Is there any API function that recognize the Experiment file?

BK
bhcho
 
Posts: 236
Joined: Mon Apr 05, 2010 2:15 pm

Re: Querying the external XML file (written by OMERO.editor)

Postby jmoore » Tue Feb 01, 2011 7:47 am

bhcho wrote:Thanks so much again.


Gladly.

1.
Can I get the value of a specific field for a specific image object?
for example, can I retrieve the value of "A.B.C1" field for an Image object of 5309? (rather than retrieve all the image IDs that has the value of "Bob" for "A.B.C1" field)


No. Lucene does not function as a key/value store, so you can't retrieve the values back out that you mapped in. (It could be made to do so, but that could well harm performance).

2.
As you know, I want to parse the Experiment (XML) file and map all the values to the Lucene Index.
I think I need to use the following part to recognize if the OriginalFile is the Experiment file or not (rather than another generic XML file).
Code: Select all
    public void set(final String name, final Object value,
            final Document document, final LuceneOptions _opts) {

        if (value instanceof OriginalFile) {
             // 1. to recognize if it's the experiment file. If it's not, do nothing from here.
             // 2. to parse the XML file
             // 3. map all the parsed values to Lucene Index
         }


Is there any API function that recognize the Experiment file?


Before getting to your points, if you use this instanceof check, then your index will only updated if the OriginalFile itself is modified, as opposed to if the file is attached to an image. For that, you'd need to check for "instanceof Image" and then examine the annotationLinks on the image itself.

The fields you'll want to examine on the OriginalFile include: name and mimetype. Where name will (probably) be of the form "*.cpe.xml" and mimetype will be "text/xml". You might also check the value of "FileAnnotation.ns" depending on how the protocol files were attached.

Cheers,
~Josh
User avatar
jmoore
Site Admin
 
Posts: 1591
Joined: Fri May 22, 2009 1:29 pm
Location: Germany

PreviousNext

Return to Developer Discussion

Who is online

Users browsing this forum: No registered users and 1 guest