Index format and compatibility Leave feedback

This page describes the index format introduced in GroupDocs.Search for .NET 26.10 and the rules that define which indexes can be opened, updated, merged and saved. Other pages refer to these rules:

• Migration notes
• Update index
• Merge indexes
• Search index location

Version 26.10 introduces a new format for storing an index on disk. Compared with previous versions, the new format takes less disk space and memory, is opened faster, protects index data with checksums and applies all changes to an index atomically: if an operation is interrupted by an error, a power failure or termination of the process, the index remains in its previous consistent state.

The new format determines the following compatibility rules:

• Version 26.10 opens only indexes of the new format. The Index constructors and the LoadIntoMemoryCompletely method throw InvalidOperationException for an index created by a previous version, including versions 26.7 and 26.8. Such an index must be updated with the IndexUpdater class first.
• Previous versions cannot open an index of version 26.10. Therefore, all applications that use the same index folder, including all nodes of a search network, must be upgraded together.

The IndexUpdater class provides two methods for checking an index folder before opening it:

• The IsLatestVersion method returns true if the folder contains an index of the current version, which can be opened without updating.
• The CanUpdateVersion method returns true if the folder contains an index of a previous version that can be updated to the current version.

Both methods return false for a folder that does not contain an index and for an empty path.

An index of a previous version is updated with the UpdateVersion method. How the index is updated depends on the version that created it:

When an index is converted without re-indexing, the indexed documents, their attributes and stored texts, the image search data, the index settings and the dictionaries are preserved. Terms longer than 80 characters are not converted, because version 26.10 does not index such terms.

The UpdateVersion method follows these rules:

• The source index is never modified. The updated index is written to the folder specified by the second parameter.
• The folder for the updated index must differ from the folder of the source index: it must not be the source folder itself, a folder that contains it or a folder inside it, also when the folder is specified by another path (for example, through a link or a mapped drive). Otherwise, the method throws ArgumentException; for a null path, it throws ArgumentNullException.
• If the folder for the updated index exists, its content is deleted before the updated index is written. If the update fails, the content may already be deleted, so do not specify a folder that contains files you need.
• The method returns a value of the VersionUpdateResult enumeration:
  • Updated — the index has been updated and written to the new folder;
  • AlreadyUpToDate — the source folder already contains an index of the current version, nothing has been done;
  • Unsupported — the source folder does not contain an index that can be updated.
• The method throws InvalidDataException if the source index is damaged and cannot be converted; in this case, nothing of the update remains in the new folder. It throws IOException or UnauthorizedAccessException if a file or a folder cannot be accessed.

After a successful update, the folder of the source index can be deleted.

The following example demonstrates how to open an index regardless of whether it was created by the current or a previous version.

C#

string indexFolder = @"c:\MyIndex\"; // The folder of an existing index
string updatedIndexFolder = @"c:\MyUpdatedIndex\"; // A new empty folder for the updated index

IndexUpdater updater = new IndexUpdater();
Index index;
if (updater.IsLatestVersion(indexFolder))
{
    // The index has the current version and can be opened as is
    index = new Index(indexFolder);
}
else if (updater.CanUpdateVersion(indexFolder))
{
    // The index of a previous version is converted or re-indexed into the new folder;
    // the source index is not changed
    VersionUpdateResult result = updater.UpdateVersion(indexFolder, updatedIndexFolder);
    if (result != VersionUpdateResult.Updated)
    {
        throw new InvalidOperationException("The index cannot be updated: " + result);
    }
    index = new Index(updatedIndexFolder);
}
else
{
    // The folder does not contain an index that can be opened or updated, so a new index is created
    index = new Index(indexFolder, true);
}

Version 26.10 introduces two index settings that affect which terms are indexed: ReplaceUnpairedSurrogates and SkipBlendedOnlyTerms. Both are enabled by default in a new index. Previous versions behaved as if both settings were disabled, so an index updated from a previous version has both settings disabled, and its documents remain consistent with their terms. See the Search index settings page for details.

The settings that define how text is divided into terms are fixed when the index is created or once it contains documents:

• The IndexType, TextStorageSettings, ReplaceUnpairedSurrogates and SkipBlendedOnlyTerms settings are fixed when the index is created; a value assigned after that is ignored.
• The alphabet, the tokenization dictionary, the character replacements and the stop words, as well as the UseStopWords and UseCharacterReplacements settings, can be changed only while the index contains no documents and is not performing an operation. Otherwise, a change throws InvalidOperationException; a call that does not change the content is allowed. Configure them once, after creating the index and before adding the first document.

The other dictionaries (synonyms, homophones, aliases, the spelling corrector and document passwords) can be changed at any time. The SearchThreads setting of an existing index can also be changed at any time through the Index.IndexSettings property; the new value is saved in the index.

The Merge method merges only indexes that divide the text of documents into terms in the same way, because a search finds a document reliably only if the document was indexed with the dictionaries and settings that the index uses. Before making any changes, the method compares the current index with every merged index that contains documents:

• the UseStopWords, UseCharacterReplacements, ReplaceUnpairedSurrogates and SkipBlendedOnlyTerms settings;
• the alphabet and the tokenization dictionary;
• the stop word dictionary, if the indexes use stop words;
• the character replacement dictionary, if the indexes use character replacements.

If a difference is found, the method throws InvalidOperationException, whose message names the differences, and no index is changed. This also happens when MergeOptions.IsAsync is true.

Other rules of merging:

• A merged index of a previous version must be updated with the IndexUpdater class before merging.
• The alphabet, the tokenization dictionary, the stop word dictionary and the character replacement dictionary of the current index are not changed by merging. The other dictionaries of the merged indexes are merged into the dictionaries of the current index.
• Indexes of another IndexType are not merged.
• A merge is not performed if a merged index is performing an operation or has failed when the merge starts; the ErrorOccurred event reports it.

An index updated from a previous version has the ReplaceUnpairedSurrogates and SkipBlendedOnlyTerms settings disabled, while a new index has them enabled by default. The following example demonstrates how to create a new index into which an updated index can be merged.

C#

string indexFolder = @"c:\MyIndex\";
string updatedIndexFolder = @"c:\MyUpdatedIndex\"; // An index updated from a previous version
string documentsFolder = @"c:\MyDocuments\";

// The new index divides text into terms as the previous versions did
IndexSettings settings = new IndexSettings();
settings.ReplaceUnpairedSurrogates = false;
settings.SkipBlendedOnlyTerms = false;

Index index = new Index(indexFolder, settings, true);
index.Add(documentsFolder);

Index updatedIndex = new Index(updatedIndexFolder);
try
{
    index.Merge(updatedIndex, new MergeOptions());
}
catch (InvalidOperationException exception)
{
    // The indexes divide text into terms differently, for example, they use different alphabets;
    // the message names the differences
    Console.WriteLine(exception.Message);
}

The SaveTo method saves an index, including an index loaded into memory, to another folder. The method follows these rules:

• The method throws InvalidOperationException if the folder is the folder of the index itself, also when it is specified differently (with a trailing separator, in a different letter case on Windows, or through a link or a mapped drive).
• An index of the current version in the target folder is replaced by the saved index. Saving can be repeated to the same folder, also after a save that failed or was interrupted.
• The method throws InvalidOperationException and does not change the folder if the folder contains an index of another version (for example, an index created by a previous version) or a damaged index. An application that regularly saves its index to the same folder, for example as a backup, has to empty that folder once after upgrading to version 26.10.

All nodes of a search network must use the same version of GroupDocs.Search. Before starting the nodes of version 26.10, update the index of each shard with the IndexUpdater class or create the shards anew.

A shard accepts another content of the alphabet, the tokenization dictionary, the stop words or the character replacements from the SetDictionary method only while its index contains no documents. Otherwise, the shard keeps its own dictionary and writes the reason to its log. See the Managing dictionaries in network page for details.

You may easily run the code from documentation articles and see the features in action in our GitHub examples:

• GroupDocs.Search for .NET examples

• GroupDocs.Search for Java examples

Along with full featured .NET library we provide simple, but powerful free Apps.

You are welcome to search over your PDF, DOC, DOCX, PPT, PPTX, XLS, XLSX and more with our free online Free Online Document Search App.