FTIntegrity is used to verify indexing after migration or to get a status after a normal indexing. The tool is a standalone Java program that verifies all types that are registered in the dmi_registry_table with the user dm_fulltext_index_user. By default, the utility compares the object ID and i_vstamp between the repository and xPlore.

In this blog I will share with you the default FTIntegrity capabilities, and also some optional configurations.

Default Configuration

Preparation

A script is given to execute the FTIntegrity Tool, but some preparation are needed before the first use:

• Verify that the IndexAgent is started
• Define the docbase name:

  export DCTM_DOCBASE_NAME="MYREPO"

• Navigate to XPLORE_HOME/setup/indexagent/tools
• Substitute the repository instance owner password in the script ftintegrity_for_${DCTM_DOCBASE_NAME}.sh, but of course for security reasons it is highly recommanded to don’t put the password into the script, the workaround is to store the password:

  stty -echo;read MDP;stty echo

• Update the script to take the password as a parameter:

  sed -i 's/password_change_me/$1/g' ftintegrity_for_${DCTM_DOCBASE_NAME}.sh

Execution

Now you can start the FT Integrity tool, usually it takes long time so it is recommended to execute it with nohup to continue running even if the session is disconnected:

nohup ./ftintegrity_for_${DCTM_DOCBASE_NAME}.sh $MDP > ftintegrity_for_${DCTM_DOCBASE_NAME}_$(date +%Y%m%d).log &

Result

Wait the end of the execution, then check files generated under XPLORE_HOME/setup/indexagent/tools:

-rw-r-----. 1 xplore xplore    15640 Dec  3 09:22 ObjectId-indexOnly.txt
-rw-r-----. 1 xplore xplore     7734 Dec  3 09:25 ObjectId-common-version-mismatch.txt
-rw-r-----. 1 xplore xplore  4237982 Dec  3 09:25 ObjectId-common-version-match.txt
-rw-r-----. 1 xplore xplore 20715741 Dec  3 09:25 ObjectId-dctmOnly.txt
-rw-r-----. 1 xplore xplore     1267 Dec  3 09:25 ftintegrity_for_MYREPO_20191203.log

The script generates four reports:

• ObjectId-common-version-match.txt
  This file contains the object IDs and i_vstamp values of all objects in the index and the repository and having identical i_vstamp values in both places.
• ObjectId-common-version-mismatch.txt
  This file records all objects in the index and the repository with identical object IDs but nonmatching i_vstamp values. For each object, it records the objectID, i_vstamp value in the repository, and i_vstamp value in the index. The mismatch is on objects that were modified during or after migration.
• ObjectId-dctmOnly.txt
  This report contains the object IDs and i_vstamp values of objects in the repository but not in the index. The objects in this report could be documents that failed indexing, documents that were filtered out, or new objects generated after migration.
• ObjectId-indexOnly.txt
  This report contains the object IDs and i_vstamp values of objects in the index but not in the repository.
  These objects were removed from the repository during or after migration, before the event has updated the index.

You can resubmit lists (version-mismatch, dctmOnly, and indexOnly) to align the index with the docbase. To do so start the IndexAgent in normal mode, then click Object File and browse to the file.

Optional Configuration

Use filter

As you can see below the dctm Only is huge so it contains a lot of documents:

-rw-r-----. 1 xplore xplore 20715741 Dec  3 09:25 ObjectId-dctmOnly.txt

This is due to the fact that some filters are configured in the Docbase, and avoid indexing some documents, to check these filters execute the below API query:

?,c,select r_object_id,object_name,primary_class from dmc_module where any a_interfaces='com.documentum.fc.indexagent.IDfCustomIndexFilter'

In my case I get the below result:

r_object_id				object_name																			primary_class
----------------		----------------------------------------------------------------------------		----------------------------------------------------------------------------------
0b01e24080000883		com.documentum.services.message.impl.type.MailMessageChildFilter					com.documentum.services.message.impl.type.MailMessageChildFilter
0b01e24080000884		com.documentum.services.message.impl.type.MailMessageChildFilter..J5_D65			com.documentum.services.message.impl.type.MailMessageChildFilter
0b01e24080000bef		com.documentum.server.impl.fulltext.indexagent.filter.defaultCabinetFilterAction	com.documentum.server.impl.fulltext.indexagent.filter.defaultCabinetFilterAction
0b01e24080000bf0		com.documentum.server.impl.fulltext.indexagent.filter.defaultFolderFilterAction		com.documentum.server.impl.fulltext.indexagent.filter.defaultFolderFilterAction
0b01e24080000bf3		com.documentum.server.impl.fulltext.indexagent.filter.defaultTypeFilterAction		com.documentum.server.impl.fulltext.indexagent.filter.defaultTypeFilterAction
(5 rows affected)

By default the FTIntegrity don’t take in account the filters, to include the filters update the script and add “-useFilter T”, use the below command:

sed -i 's//& -useFilter T/' ftintegrity_for_${DCTM_DOCBASE_NAME}.sh

Check specific type

Use the option -checkType to check a specific object type, use the below command to update:

sed -i '/BATCH_SIZE/s/$/ -checkType dm_document/' ftintegrity_for_${DCTM_DOCBASE_NAME}.sh

In this way, only dm_document are verified.

Compare specific metadata

You can compare metadata values which compares object IDs and the specified attributes, to do so use the option -checkMetadata. Be careful, when this option is used you have to add -checkType option, for example, to compare only the a_is_hidden attribute:
Define the list of attributes in a file (one attribute by line):

cat $XPLORE_HOME/setup/indexagent/tools/doclist.txt
a_is_hidden

Then specify the list of metadata to be checked by the script:

...
... $DSEARCH_DOMAIN $OUTPUT_FILEPATH $BATCH_SIZE -useFilter T -checkType dm_document -CheckMetadata XPLORE_HOME/setup/indexagent/tools/doclist.txt

As a result, you will have two new reports:

• Object-Metadata-mismatch.txt
  Contains all the objects with metadata that has inconsistencies.
• Object-Metadata-match.txt
  Contains all the objects with metadata that has valid consistencies.

In this way, you can compare really everything! Don’t hesitate to ask or share your experience with this tool 😉

Post Views: 1,993