Lucene Search

Search Strategies

There are searches – and searches.

• One can search by simply filling in words in the query field and press return (or click the magnifying glass).
• One can use the standard Lucene search options. These consists of marking which words you would like to occur as hits and which word must (or must not) occur, of stringing together words with AND, OR and NOT, grouping words with parentheses, and so on. Quite complicated searches can be made with these options.
• One can use Lucene wildcards - ? for any single character and * for any sequence of zero, one or more characters.
• Even more advanced, one can use regular expressions, the most precise (and complicated) way of searching for texts strings.

All of these search options can be used in combination, so one can freely e.g. mix plain, wildcard and regular expression searches.

These three different search strategies will be presented below, after a brief description of the way hits are displayed.

Hitlist and Search Relevance

When searching for books in a library catalogue, one can usually choose to have the hits displayed according to relevance or according to author, title or suchlike. In the SARIT app, hits from Lucene searches are only displayed according to relevance, according to a "score" computed for each search. The score is a quite complicated thing in itself, but basically, the more times your search terms occur in your search scope and the less common they are in the index (that is, in SARIT works) the higher the score they will get and the more prominent they will be in the hitlist.

Standard Lucene Syntax

In the following, examples are drawn from imaginary searches in Shakespeare's works. It is intended that somone knowledgable in Sanskrit convert the examples to something more relavant to SARIT.

If you simply fill in some words in the query field, and do not use any of the special Lucene search operators, you are saying that you would like to see as many of the words in the search scope as possible.

With Lucene standard syntax, there are two ways you can impose restrictions of the co-occurrence of search terms: you can either prefix words with + and - or you use the boolean terms AND, OR and NOT (written in upper-case). In both cases, you can additionally group your search expressions using parentheses.

The option of using + and - is better suited to a search which orders hits according to score. Here you let words stand as they are (without + or -) if you would like all of them to occur in hits, but if there is only one of them present, you also want to have it displayed as a hit If you prefix one or more words with +, they must occur in a hit, and if you prefix them with -, they must not occur as a hit. If you search for snake fillet you get a lot of hits with either snake or killed and perhaps some with both. If you search for "snake +fillet, all your hits will contain fillet, but they may or may not contain snake. If you search for snake -fillet, you would get hits with snake, but only if they do not contain fillet.

If you use AND, OR and NOT, the logic is rather different. If you search for snake AND fillet you get hits with both snake and fillet and none with only one of them. This corresponds to +snake +killed. If you search for snake OR killed, this is the same as simply searching for snake fillet. If you search for snake NOT fillet, this equals searching for snake -fillet.

Searches can acquire higher complexity use of parentheses. Here the use of AND, OR and NOT may come more naturally. Say you want to find passages where the word fillet occurs but where also at least one of the words snake, deer, bird occurs. You can express this by (snake OR deer OR bird) AND fillet. An AND enforces "must occur" on both sides, so both one of the animals and the word fillet have to occur in the hits. Say (for some reason) you do not wish the words pricket and mouse to occur in your hits – you then embroider your search expression with NOT (pricket OR mouse) as (snake OR deer OR bird) AND fillet NOT (pricket OR mouse)

If you simply search for pricket OR deer AND killed, you will (because the AND rubs off to the left), search for passages where deer and fillet must occur, but you would also like pricket to be marked as a hit. You can enforce a certain logic on your query by grouping with parentheses.

If you search for (snake OR deer) AND fillet you are saying that one or both of snake and deer must occur, as must fillet.

If you search for snake OR (deer AND fillet), you would like to retrieve hits where snake occurs and you would like to retrieve hits where deer and fillet go together. In practice this means that you will get a lot of snake-only hits.

You can also nest parentheses, e.g. (snake OR (deer AND fillet)) NOT pricket will remove the hits with pricket from snake OR (deer AND fillet.

As you can see, the options are many …. And as if this was not enough, there is also regex – and regex syntax combined with standard syntax!

Phrase Search

A phrase search searches for a series of words in sequence. You activate phrase search by enclosing the words in quotations marks. If you want to search for all occurences of "fenny snake", you should input the search expression "fenny snake". This is the way searches are performed in word processing documents, except that here punctuation is disregarded. To find "Fire burn, and cauldron bubble", you do not have to input the comma (but it does not harm - it is removed automatically from your search expression).

Proximity Search

A proximity search allows you to search for all of the words in the hits within the search scope, in the order they occur in, but within a certain proximity. You put the search engine in proximity search mode by enclosing the words in quotation marks and adding a tilde ~ after the rightmost quotation mark, immediately followed by the number of words you wish to allow between the two words. If you search for "Fire bubble"~3 you will thus find "Fire burn, and cauldron bubble", but if you only allow two words to intervene, "Fire bubble"~2, you will not. More than two words are allowed: you can retrieve "I pray thee, stay with us: go not to Wittenberg" with "pray stay Wittenberg"~6.

If you wish to search for words within a certain proximity, but regardless of the order they occur in, you have to use an OR expression: "slumber beware"~5 OR "beware slumber"~5.

Fuzzy Search

"Fuzzy Search" needs a little explanation. If you take a word, like "snake", you can make changes and additions to it. One change would thus give you "spake", "slave", "snare" and "snakes". If you make one more edits based on this, you can easily see that a lot of words can be generated. Since this search is very time-consuming, the maximum number of "edits" you can make is 2. To activate fuzzy search, you add a tilde ~ after the word in question, snake~2.

Wildcard Expressions

Wildcard Expressions allow you to mask individual characters or sequences of characters inside words. Say, to switch examples, you wish to find all occurrences of "test" and of "text". Now, you could search for text test (or text OR test), and this would give you what you wanted, but you can also use the wildcard ? and search for te?t instead. This will find all words that consists of the two character "te", plus one character which can be anything, and end up with the character "t". It would also find "teat".

In addition to ? which requires one and one only character, you can also use * which stands for any character zero, one or more times. If you search for te*t, you are therefore likely to find many more words, among them "test" and "text", but also "tempest", "testament" and "tent"

You do not have to signal in any way that you are performing a wildcard search: just including ? or * is enough.

A wildcard character cannot occur first in a search expression, so you cannot find "Hamlet" with ?amlet or *let. If you wish to perform searches of this kind, you should use a regular expression.

Regular Expressions

Regular Expressions are also known as "regex" or "regexp". They are a very powerful tool for searching text (and for replacing text, but this is not relevant here). Lucene only supports a restricted range of regex operators, but they should be sufficient for most uses.

You put the search engine into regex mode by enclosing your search term with slashes. So you would search e.g. for /.{3}let/ to find "Hamlet" (but also, e.g. "fillet").