Full Text Index

Using match="..."

In addition to defining an index on a given qualified name, you can also specify a "path" with wildcards. This feature might be subject to change, so please be careful when using it.

Assume you want to define an index on all the possible elements below <SPEECH>. You can do this by creating one index for every element:

<text qname="LINE"/>
<text qname="SPEAKER"/>

As a shortcut, you can use a match attribute with a wildcard:

<text match="//SPEECH/*"/>

This will create a separate index on each child element of SPEECH it encounters. Please note that the argument to match is a simple path pattern, not a full XPath expression. For the time being, it only allows:

• / and // to denote child or descendant steps,

• * wildcard selector to match an arbitrary element,

• matching a single attribute's value, e.g. foo[@bar = 'xyz']

As explained above, you have to figure out which parts of your document will likely be interesting as context for a full text query. The full text index works best if the context isn't too narrow. For example, if you have a document structure with section <div>s, headings and paragraphs, you would probably want to create an index on the <div>s and maybe on the headings, so the user can differentiate between the two.

In some cases, you could decide to put the index on the paragraph level. Then you don't need the index on the section, since you can always get from the paragraph back to the section.

If you query a larger context, you can use the KWIC module to show the user text surrounding each match. Or you can ask eXist-db to highlight each match with an <exist:match> tag, which you can later use to locate the matches within the text.

Whitespace Treatment and Ignored Content

<size><width>12</width><height>8</height></size>

<p>This is <b>un</b>clear.</p>

<inline qname="b"/>

<text qname="p">
    <inline qname="em"/>
</text>

<p>This is a paragraph
<note>containing an inline note</note>.</p>

<ignore qname="note"/>

<lucene>
    <text qname="p"/>
    <text qname="note"/>
    <ignore qname="note"/>
</lucene>

//p[ft:query(., "note")]

//p[ft:query(note, "note")]

Boost

A boost value can be assigned to an index to give it a higher score. The score for each match will be multiplied by the boost factor (default is: 1.0). For example, you may want to rank matches in titles higher than other matches.

Here's how to configure the documentation search indexes in eXist-db:

<lucene>
    <analyzer class="org.apache.lucene.analysis.standard.StandardAnalyzer"/>
    <text qname="section">
        <ignore qname="title"/>
        <ignore qname="programlisting"/>
        <ignore qname="screen"/>
        <ignore qname="synopsis"/>
    </text>
    <text qname="para"/>
    <text qname="title" boost="2.0"/>
    <ignore qname="title"/>
</lucene>

The <title> index gets a boost of 2.0 to make sure that its matches get a higher score. Since the <title> element occurs within <section>, we add an ignore rule to the index definition on the section and create a separate index on title. We also ignore titles occurring inside paragraphs. Without this, title would be matched two times.

Because the title is now indexed separately, we need to query it explicitly. For example, to search the section and the title at the same time, one could issue the following query:

for $sect in /book//section[ft:query(., "ngram")] | /book//section[ft:query(title, "ngram")]
order by ft:score($sect) descending 
return $sect

<text qname="@val">
    <match-sibling-attr boost="25" qname="att" value="writtenForm"/>
</text>

<text qname="feat">
    <has-attr boost="0" qname="xml:lang"/>
</text>

Analyzers

One of the strengths of Lucene is that it allows the developer to determine nearly every aspect of text analysis. This is done through analyzer classes, which combine a tokenizer with a chain of filters to post-process the tokenized text. eXist-db's Lucene module already allows different analyzers to be used for different indexes.

<lucene>
    <analyzer class="org.apache.lucene.analysis.standard.StandardAnalyzer"/>
    <analyzer id="ws" class="org.apache.lucene.analysis.core.WhitespaceAnalyzer"/>
    <text match="//SPEECH//*"/>
    <text qname="TITLE" analyzer="ws"/>
</lucene>

In the example above, we define that Lucene's StandardAnalyzer should be used by default (the <analyzer> element without id attribute). We provide an additional analyzer and assign it the id ws, by which the analyzer can be referenced in the actual index definitions.

The whitespace analyzer is the most basic one. As the name implies, it tokenizes the text at white space characters, but treats all other characters - including punctuation - as part of the token. The tokens are not converted to lower case and there's no stopword filter applied.

<analyzer id="stdstops" class="org.apache.lucene.analysis.standard.StandardAnalyzer">
    <param name="stopwords" type="java.io.FileReader" value="/tmp/stop.txt"/>
</analyzer>

<analyzer id="stdstops" class="org.apache.lucene.analysis.standard.StandardAnalyzer">
    <param name="stopwords" type="org.apache.lucene.analysis.util.CharArraySet">
        <value>the</value>
        <value>this</value>
        <value>and</value>
        <value>that</value>
    </param>
</analyzer>

<analyzer id="sbstops" class="org.apache.lucene.analysis.snowball.SnowballAnalyzer">
    <param name="name" value="English"/>
    <param name="stopwords" type="org.apache.lucene.analysis.util.CharArraySet">
        <value>the</value>
        <value>this</value>
        <value>and</value>
        <value>that</value>
    </param>
</analyzer>

Defining Fields

Sometimes you want to define different Lucene indexes on the same set of elements, for instance to use a different analyzer. eXist-db allows to name a certain index using the field attribute:

<text field="title" qname="title" analyzer="en"/>

Such an index is called named index. See Query a Named Index on how to query these indexes.

Query a Named Index

To query a named index (see Defining Fields), use the ft:query-field($fieldName, $query) instead of ft:query:

ft:query-field("title", "xml")

ft:query-field works exactly like ft:query, except that the set of nodes to search is determined by the nodes in the named index. The function returns the nodes selected by the query, which would be <title> elements in the example above.

You can use ft:query-field with an XPath filter expression, just as you would call ft:query:

//section[ft:query-field("title", "xml")]

Describing Queries in XML

Lucene's default query syntax does not provide access to all available features. However, eXist-db's ft:query function also accepts a description of the query in XML, as an alternative to passing a query string. The XML description closely mirrors Lucene's query API. It is transformed into an internal tree of query objects, which is directly passed to Lucene for execution. This has several advantages, for example you can specify if the order of terms should be relevant for a phrase query:

let $query :=
    <query>
        <near ordered="no">miserable nation</near>
    </query>
return
    //SPEECH[ft:query(., $query)]

The following elements may occur within a query description:

<term>

Defines a single term to be searched in the index. If the root query element contains a sequence of term elements, wrap them in <bool/> and they will be combined as in a boolean "or" query. For example:

let $query :=
    <query>
        <bool><term>nation</term><term>miserable</term></bool>
    </query>
return
//SPEECH[ft:query(., $query)]

This finds all <SPEECH> elements containing either nation or miserable or both.

<wildcard>

A string with a * wildcard in it. This will be matched against the terms of a document. Can be used instead of a <term> element. For example:

let $query :=
    <query>
        <bool><term>nation</term><wildcard>miser*</wildcard></bool>
    </query>
return
//SPEECH[ft:query(., $query)]

<regex>

A regular expression which will be matched against the terms of a document. Can be used instead of a <term> element. For example:

let $query :=
    <query>
        <bool><term>nation</term><regex>miser.*</regex></bool>
    </query>
return
//SPEECH[ft:query(., $query)]

<bool>

Constructs a boolean query from its children. Each child element may have an occurrence indicator, which could be either must, should or not:

must

this part of the query must be matched

should

this part of the query should be matched, but doesn't need to

not

this part of the query must not be matched

For instance:

let $query :=
    <query>
        <bool><term occur="must">boil</term><term occur="should">bubble</term></bool>
    </query>
return //SPEECH[ft:query(LINE, $query)]

<phrase>

Searches for a group of terms occurring in the correct order. The element may either contain explicit <term> elements or text content. Text will be automatically tokenized into a sequence of terms. For example:

let $query :=
    <query>
        <phrase>cauldron boil</phrase>
    </query>
return //SPEECH[ft:query(., $query)]

This has the same effect as:

let $query :=
    <query>
        <phrase><term>cauldron</term><term>boil</term></phrase>
    </query>
return //SPEECH[ft:query(., $query)]

The attribute slop can be used for a proximity search: Lucene will try to find terms which are within the specified distance:

let $query :=
    <query>
        <phrase slop="10"><term>frog</term><term>dog</term></phrase>
    </query>
return //SPEECH[ft:query(., $query)]

<near>

<near> is a powerful alternative to <phrase> and one of the features not available through the standard Lucene query parser.

If the element has text content only, it will be tokenized into terms and the expression behaves like <phrase>. Otherwise it may contain any combination of <term>, <first> and nested <near> elements. This makes it possible to search for two sequences of terms which are within a specific distance. For example:

let $query :=
    <query>
        <near slop="20"><term>snake</term><near slop="1">tongue dog</near></near>
    </query>
return //SPEECH[ft:query(., $query)]

Element <first> matches a span against the start of the text in the context node. It takes an optional attribute end to specify the maximum distance from the start of the text. For example:

let $query :=
    <query>
        <near slop="50"><first end="2"><near>second witch</near></first><near
slop="1">tongue dog</near></near>
    </query>
    return //SPEECH[ft:query(., $query)]

As shown above, the content of <first> can again be text, a <term> or <near>.

Contrary to <phrase>, <near> can be told to ignore the order of its components. Use parameter ordered="yes|no" to change near's behaviour. For example:

let $query :=
    <query>
        <near slop="100" ordered="no"><term>bubble</term><term>fillet</term></near>
    </query>
return //SPEECH[ft:query(., $query)]

All elements in a query may have an optional boost parameter (float). The score of the nodes matching the corresponding query part will be multiplied by this factor.

Additional parameters

The ft:query function allows a third parameter for passing additional settings to the query engine. This parameter must be an XML fragment which lists the configuration properties to be set as child elements:

let $options :=
    <options>
        <default-operator>and</default-operator>
        <phrase-slop>1</phrase-slop>
        <leading-wildcard>no</leading-wildcard>
        <filter-rewrite>yes</filter-rewrite>
    </options>
return
    //SPEECH[ft:query(., $query, $options)]

The meaning of those properties is as follows

filter-rewrite

Controls how terms are expanded for wildcard or regular expression searches. If set to yes, Lucene will use a filter to pre-process matching terms. If set to no, all matching terms will be added to a single boolean query which is then executed. This may generate a "too many clauses" exception when applied to large data sets. Setting filter-rewrite to yes avoids those issues.

default-operator

The default operator with which multiple terms will be combined. Allowed values: or, and.

phrase-slop

Sets the default slop for phrases. If 0, then exact phrase matches are required. Default value is 0.

leading-wildcard

When set to yes, * or ? are allowed as the first character of a PrefixQuery and WildcardQuery. Note that this can produce very slow queries on big indexes.