Index format and compatibility Leave feedback

This page describes the index format introduced in GroupDocs.Search 26.10 and the rules that define which indexes can be opened, updated, merged and saved. Other pages refer to these rules:

• Migration notes
• Update index
• Merge indexes
• Search index location

Version 26.10 introduces a new format for storing an index on disk. Compared with previous versions, the new format takes less disk space and memory, is opened faster, protects index data with checksums and applies all changes to an index atomically: if an operation is interrupted by an error, a power failure or termination of the process, the index remains in its previous consistent state.

The new format determines the following compatibility rules:

• Version 26.10 opens only indexes of the new format. The Index constructors and the load_into_memory_completely method raise RuntimeError for an index created by a previous version, including versions 26.7 and 26.8. Such an index must be updated with the IndexUpdater class first.
• Previous versions cannot open an index of version 26.10. Therefore, all applications that use the same index folder, including all nodes of a search network, must be upgraded together.

The IndexUpdater class provides two methods for checking an index folder before opening it:

• The is_latest_version method returns True if the folder contains an index of the current version, which can be opened without updating.
• The can_update_version method returns True if the folder contains an index of a previous version that can be updated to the current version.

Both methods return False for a folder that does not contain an index and for an empty path.

An index of a previous version is updated with the update_version method. How the index is updated depends on the version that created it:

When an index is converted without re-indexing, the indexed documents, their attributes and stored texts, the image search data, the index settings and the dictionaries are preserved. Terms longer than 80 characters are not converted, because version 26.10 does not index such terms.

The update_version method follows these rules:

• The source index is never modified. The updated index is written to the folder specified by the second parameter.
• The folder for the updated index must differ from the folder of the source index: it must not be the source folder itself, a folder that contains it or a folder inside it, also when the folder is specified by another path (for example, through a link or a mapped drive). Otherwise, the method raises ValueError; for a None path, it also raises ValueError.
• If the folder for the updated index exists, its content is deleted before the updated index is written. If the update fails, the content may already be deleted, so do not specify a folder that contains files you need.
• The method returns a value of the VersionUpdateResult enumeration:
  • UPDATED — the index has been updated and written to the new folder;
  • ALREADY_UP_TO_DATE — the source folder already contains an index of the current version, nothing has been done;
  • UNSUPPORTED — the source folder does not contain an index that can be updated.
• The method raises GroupDocsSearchException if the source index is damaged and cannot be converted; in this case, nothing of the update remains in the new folder. It raises PermissionError or GroupDocsSearchException if a file or a folder cannot be accessed.

After a successful update, the folder of the source index can be deleted.

The following example demonstrates how to open an index regardless of whether it was created by the current or a previous version. To print the name of a result, the example passes it to VersionUpdateResult.

from groupdocs.search import Index, IndexUpdater
from groupdocs.search.common import VersionUpdateResult

index_folder = "./old-index"  # The folder of an existing index, here one created by a previous version
updated_index_folder = "./updated-index"  # A new empty folder for the updated index

updater = IndexUpdater()
if updater.is_latest_version(index_folder):
    # The index has the current version and can be opened as is
    index = Index(index_folder)
    print("The index has the current version.")
elif updater.can_update_version(index_folder):
    # The index of a previous version is converted or re-indexed into the new folder;
    # the source index is not changed
    result = updater.update_version(index_folder, updated_index_folder)
    if result != VersionUpdateResult.UPDATED:
        raise RuntimeError("The index cannot be updated: " + result.name)
    index = Index(updated_index_folder)
    print("The index has been updated.")
else:
    # The folder does not contain an index that can be opened or updated, so a new index is created
    index = Index(index_folder, True)
    print("A new index has been created.")

print("Indexed documents:", len(index.get_indexed_documents()))

The index has been updated.
Indexed documents: 4

Version 26.10 introduces two index settings that affect which terms are indexed: replace_unpaired_surrogates and skip_blended_only_terms. Both are enabled by default in a new index. Previous versions behaved as if both settings were disabled, so an index updated from a previous version has both settings disabled, and its documents remain consistent with their terms. See the Search index settings page for details.

The settings that define how text is divided into terms are fixed when the index is created or once it contains documents:

• The index_type, text_storage_settings, replace_unpaired_surrogates and skip_blended_only_terms settings are fixed when the index is created; a value assigned after that is ignored.
• The alphabet, the tokenization dictionary, the character replacements and the stop words, as well as the use_stop_words and use_character_replacements settings, can be changed only while the index contains no documents and is not performing an operation. Otherwise, a change raises RuntimeError; a call that does not change the content is allowed. Configure them once, after creating the index and before adding the first document.

The other dictionaries (synonyms, homophones, aliases, the spelling corrector and document passwords) can be changed at any time. The search_threads setting of an existing index can also be changed at any time through the Index.index_settings property; the new value is saved in the index.

The merge method merges only indexes that divide the text of documents into terms in the same way, because a search finds a document reliably only if the document was indexed with the dictionaries and settings that the index uses. Before making any changes, the method compares the current index with every merged index that contains documents:

• the use_stop_words, use_character_replacements, replace_unpaired_surrogates and skip_blended_only_terms settings;
• the alphabet and the tokenization dictionary;
• the stop word dictionary, if the indexes use stop words;
• the character replacement dictionary, if the indexes use character replacements.

If a difference is found, the method raises RuntimeError, whose message names the differences, and no index is changed. This also happens when MergeOptions.is_async is True.

Other rules of merging:

• A merged index of a previous version must be updated with the IndexUpdater class before merging.
• The alphabet, the tokenization dictionary, the stop word dictionary and the character replacement dictionary of the current index are not changed by merging. The other dictionaries of the merged indexes are merged into the dictionaries of the current index.
• Indexes of another IndexType are not merged.
• A merge is not performed if a merged index is performing an operation or has failed when the merge starts; the error_occurred event reports it.

An index updated from a previous version has the replace_unpaired_surrogates and skip_blended_only_terms settings disabled, while a new index has them enabled by default. The following example demonstrates how to create a new index into which an updated index can be merged. The example prints the exception’s message attribute, the message alone; str(exception) also contains the .NET stack trace.

from groupdocs.search import Index, IndexSettings
from groupdocs.search.options import MergeOptions

# The new index divides text into terms as the previous versions did
settings = IndexSettings()
settings.replace_unpaired_surrogates = False
settings.skip_blended_only_terms = False

index = Index("./index", settings, True)
index.add("./documents")

updated_index = Index("./updated-index")  # An index updated from a previous version
try:
    index.merge(updated_index, MergeOptions())
    print("Indexed documents after merging:", len(index.get_indexed_documents()))
except RuntimeError as exception:
    # The indexes divide text into terms differently, for example, they use different alphabets;
    # the message names the differences
    print(exception.message)

Indexed documents after merging: 7

The save_to method saves an index, including an index loaded into memory, to another folder. The method follows these rules:

• The method raises RuntimeError if the folder is the folder of the index itself, also when it is specified differently (with a trailing separator, in a different letter case on Windows, or through a link or a mapped drive).
• An index of the current version in the target folder is replaced by the saved index. Saving can be repeated to the same folder, also after a save that failed or was interrupted.
• The method raises RuntimeError and does not change the folder if the folder contains an index of another version (for example, an index created by a previous version) or a damaged index. An application that regularly saves its index to the same folder, for example as a backup, has to empty that folder once after upgrading to version 26.10.

All nodes of a search network must use the same version of GroupDocs.Search. Before starting the nodes of version 26.10, update the index of each shard with the IndexUpdater class or create the shards anew.

A shard accepts another content of the alphabet, the tokenization dictionary, the stop words or the character replacements from the set_dictionary method only while its index contains no documents. Otherwise, the shard keeps its own dictionary and writes the reason to its log. See the Managing dictionaries in network page for details.

You may easily run the code from documentation articles and see the features in action in our GitHub examples:

• GroupDocs.Search for Python via .NET examples

Along with the full-featured library, we provide simple but powerful free apps.

You are welcome to search over your PDF, DOC, DOCX, PPT, PPTX, XLS, XLSX and more with our free online Free Online Document Search App.