{"id":13137,"date":"2019-12-03T18:00:55","date_gmt":"2019-12-03T17:00:55","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/"},"modified":"2019-12-03T18:00:55","modified_gmt":"2019-12-03T17:00:55","slug":"documentum-ft-ftintegrity-tool","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/","title":{"rendered":"Documentum &#8211; FT &#8211; FTIntegrity Tool"},"content":{"rendered":"<p>FTIntegrity is used to verify indexing after migration or to get a status after a normal indexing. The tool is a standalone Java program that verifies all types that are registered in the dmi_registry_table with the user dm_fulltext_index_user. By default, the utility compares the object ID and i_vstamp between the repository and xPlore.<br \/>\n<!--more--><\/p>\n<p>In this blog I will share with you the default FTIntegrity capabilities, and also some optional configurations. <\/p>\n<h1> Default Configuration <\/h1>\n<h2>Preparation<\/h2>\n<p>A script is given to execute the FTIntegrity Tool, but some preparation are needed before the first use:<\/p>\n<ul>\n<li>Verify that the IndexAgent is started<\/li>\n<li>Define the docbase name:\n<pre class=\"brush: bash; gutter: true; first-line: 1\">export DCTM_DOCBASE_NAME=\"MYREPO\"<\/pre>\n<\/li>\n<li>Navigate to XPLORE_HOME\/setup\/indexagent\/tools<\/li>\n<li>Substitute the repository instance owner password in the script ftintegrity_for_${DCTM_DOCBASE_NAME}.sh, but of course for security reasons it is highly recommanded to don&#8217;t put the password into the script, the workaround is to store the password:\n<pre class=\"brush: bash; gutter: true; first-line: 1\">stty -echo;read MDP;stty echo<\/pre>\n<\/li>\n<li>Update the script to take the password as a parameter:\n<pre class=\"brush: bash; gutter: true; first-line: 1\">sed -i 's\/password_change_me\/$1\/g' ftintegrity_for_${DCTM_DOCBASE_NAME}.sh<\/pre>\n<\/li>\n<\/ul>\n<h2>Execution<\/h2>\n<p>Now you can start the FT Integrity tool, usually it takes long time so it is recommended to execute it with nohup to continue running even if the session is disconnected:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\nnohup .\/ftintegrity_for_${DCTM_DOCBASE_NAME}.sh $MDP &gt; ftintegrity_for_${DCTM_DOCBASE_NAME}_$(date +%Y%m%d).log &amp;\n<\/pre>\n<h2>Result<\/h2>\n<p>Wait the end of the execution, then check files generated under XPLORE_HOME\/setup\/indexagent\/tools:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\n-rw-r-----. 1 xplore xplore    15640 Dec  3 09:22 ObjectId-indexOnly.txt\n-rw-r-----. 1 xplore xplore     7734 Dec  3 09:25 ObjectId-common-version-mismatch.txt\n-rw-r-----. 1 xplore xplore  4237982 Dec  3 09:25 ObjectId-common-version-match.txt\n-rw-r-----. 1 xplore xplore 20715741 Dec  3 09:25 ObjectId-dctmOnly.txt\n-rw-r-----. 1 xplore xplore     1267 Dec  3 09:25 ftintegrity_for_MYREPO_20191203.log\n<\/pre>\n<p>The script generates four reports:<\/p>\n<ul>\n<li><strong>ObjectId-common-version-match.txt<\/strong><br \/>\nThis file contains the object IDs and i_vstamp values of all objects in the index and the repository and having identical i_vstamp values in both places.<\/li>\n<li><strong>ObjectId-common-version-mismatch.txt<\/strong><br \/>\nThis file records all objects in the index and the repository with identical object IDs but nonmatching i_vstamp values. For each object, it records the objectID, i_vstamp value in the repository, and i_vstamp value in the index. The mismatch is on objects that were modified during or after migration.<\/li>\n<li><strong>ObjectId-dctmOnly.txt<\/strong><br \/>\nThis report contains the object IDs and i_vstamp values of objects in the repository but not in the index. The objects in this report could be documents that failed indexing, documents that were filtered out, or new objects generated after migration.<\/li>\n<li><strong>ObjectId-indexOnly.txt<\/strong><br \/>\nThis report contains the object IDs and i_vstamp values of objects in the index but not in the repository.<br \/>\nThese objects were removed from the repository during or after migration, before the event has updated the index.<\/li>\n<p><em>You can resubmit lists (version-mismatch, dctmOnly, and indexOnly) to align the index with the docbase. To do so start the IndexAgent in normal mode, then click Object File and browse to the file.<\/em><\/p>\n<h1>Optional Configuration<\/h1>\n<h2> Use filter <\/h2>\n<p>As you can see below the dctm Only is huge so it contains a lot of documents:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\n-rw-r-----. 1 xplore xplore 20715741 Dec  3 09:25 ObjectId-dctmOnly.txt\n<\/pre>\n<p>This is due to the fact that some filters are configured in the Docbase, and avoid indexing some documents, to check these filters execute the below API query:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\n?,c,select r_object_id,object_name,primary_class from dmc_module where any a_interfaces='com.documentum.fc.indexagent.IDfCustomIndexFilter'\n<\/pre>\n<p>In my case I get the below result:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\nr_object_id\t\t\t\tobject_name\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\tprimary_class\n----------------\t\t----------------------------------------------------------------------------\t\t----------------------------------------------------------------------------------\n0b01e24080000883\t\tcom.documentum.services.message.impl.type.MailMessageChildFilter\t\t\t\t\tcom.documentum.services.message.impl.type.MailMessageChildFilter\n0b01e24080000884\t\tcom.documentum.services.message.impl.type.MailMessageChildFilter..J5_D65\t\t\tcom.documentum.services.message.impl.type.MailMessageChildFilter\n0b01e24080000bef\t\tcom.documentum.server.impl.fulltext.indexagent.filter.defaultCabinetFilterAction\tcom.documentum.server.impl.fulltext.indexagent.filter.defaultCabinetFilterAction\n0b01e24080000bf0\t\tcom.documentum.server.impl.fulltext.indexagent.filter.defaultFolderFilterAction\t\tcom.documentum.server.impl.fulltext.indexagent.filter.defaultFolderFilterAction\n0b01e24080000bf3\t\tcom.documentum.server.impl.fulltext.indexagent.filter.defaultTypeFilterAction\t\tcom.documentum.server.impl.fulltext.indexagent.filter.defaultTypeFilterAction\n(5 rows affected)\n<\/pre>\n<p>By default the FTIntegrity don&#8217;t take in account the filters, to include the filters update the script and add &#8220;-useFilter T&#8221;, use the below command:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\nsed -i 's\/\/&amp; -useFilter T\/' ftintegrity_for_${DCTM_DOCBASE_NAME}.sh\n<\/pre>\n<h2> Check specific type <\/h2>\n<p>Use the option -checkType to check a specific object type, use the below command to update:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\nsed -i '\/BATCH_SIZE\/s\/$\/ -checkType dm_document\/' ftintegrity_for_${DCTM_DOCBASE_NAME}.sh\n<\/pre>\n<p>In this way, only dm_document are verified.<\/p>\n<h2> Compare specific metadata <\/h2>\n<p>You can compare metadata values which compares object IDs and the specified attributes, to do so use the option -checkMetadata. Be careful, when this option is used you have to add -checkType option, for example, to compare only the a_is_hidden attribute:<br \/>\nDefine the list of attributes in a file (one attribute by line):<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\ncat $XPLORE_HOME\/setup\/indexagent\/tools\/doclist.txt\na_is_hidden\n<\/pre>\n<p>Then specify the list of metadata to be checked by the script:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\n...\n... $DSEARCH_DOMAIN $OUTPUT_FILEPATH $BATCH_SIZE -useFilter T -checkType dm_document -CheckMetadata XPLORE_HOME\/setup\/indexagent\/tools\/doclist.txt\n<\/pre>\n<p>As a result, you will have two new reports:<\/p>\n<ul>\n<li><strong>Object-Metadata-mismatch.txt<\/strong><br \/>\nContains all the objects with metadata that has inconsistencies.<\/li>\n<li><strong>Object-Metadata-match.txt<\/strong><br \/>\nContains all the objects with metadata that has valid consistencies.<\/li>\n<\/ul>\n<p>In this way, you can compare really everything! Don&#8217;t hesitate to ask or share your experience with this tool \ud83d\ude09<\/p>\n","protected":false},"excerpt":{"rendered":"<p>FTIntegrity is used to verify indexing after migration or to get a status after a normal indexing. The tool is a standalone Java program that verifies all types that are registered in the dmi_registry_table with the user dm_fulltext_index_user. By default, the utility compares the object ID and i_vstamp between the repository and xPlore.<\/p>\n","protected":false},"author":46,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[197,1183],"tags":[129,1072,963,91],"type_dbi":[],"class_list":["post-13137","post","type-post","status-publish","format-standard","hentry","category-application-integration-middleware","category-big-data","tag-documentum","tag-ft","tag-ftintegrity","tag-index"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.2 (Yoast SEO v27.2) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Documentum - FT - FTIntegrity Tool - dbi Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Documentum - FT - FTIntegrity Tool\" \/>\n<meta property=\"og:description\" content=\"FTIntegrity is used to verify indexing after migration or to get a status after a normal indexing. The tool is a standalone Java program that verifies all types that are registered in the dmi_registry_table with the user dm_fulltext_index_user. By default, the utility compares the object ID and i_vstamp between the repository and xPlore.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2019-12-03T17:00:55+00:00\" \/>\n<meta name=\"author\" content=\"David Diab\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"David Diab\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/\"},\"author\":{\"name\":\"David Diab\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/deb907c3360cacdc6c7df54b4bac3c86\"},\"headline\":\"Documentum &#8211; FT &#8211; FTIntegrity Tool\",\"datePublished\":\"2019-12-03T17:00:55+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/\"},\"wordCount\":658,\"commentCount\":0,\"keywords\":[\"Documentum\",\"FT\",\"ftintegrity\",\"index\"],\"articleSection\":[\"Application integration &amp; Middleware\",\"Big Data\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/\",\"name\":\"Documentum - FT - FTIntegrity Tool - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"datePublished\":\"2019-12-03T17:00:55+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/deb907c3360cacdc6c7df54b4bac3c86\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Documentum &#8211; FT &#8211; FTIntegrity Tool\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/deb907c3360cacdc6c7df54b4bac3c86\",\"name\":\"David Diab\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/212b1b2e4650bad3116f644ab4fb4663786d94195d7685d0704c8426da088e60?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/212b1b2e4650bad3116f644ab4fb4663786d94195d7685d0704c8426da088e60?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/212b1b2e4650bad3116f644ab4fb4663786d94195d7685d0704c8426da088e60?s=96&d=mm&r=g\",\"caption\":\"David Diab\"},\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/david-diab\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Documentum - FT - FTIntegrity Tool - dbi Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/","og_locale":"en_US","og_type":"article","og_title":"Documentum - FT - FTIntegrity Tool","og_description":"FTIntegrity is used to verify indexing after migration or to get a status after a normal indexing. The tool is a standalone Java program that verifies all types that are registered in the dmi_registry_table with the user dm_fulltext_index_user. By default, the utility compares the object ID and i_vstamp between the repository and xPlore.","og_url":"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/","og_site_name":"dbi Blog","article_published_time":"2019-12-03T17:00:55+00:00","author":"David Diab","twitter_card":"summary_large_image","twitter_misc":{"Written by":"David Diab","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/"},"author":{"name":"David Diab","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/deb907c3360cacdc6c7df54b4bac3c86"},"headline":"Documentum &#8211; FT &#8211; FTIntegrity Tool","datePublished":"2019-12-03T17:00:55+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/"},"wordCount":658,"commentCount":0,"keywords":["Documentum","FT","ftintegrity","index"],"articleSection":["Application integration &amp; Middleware","Big Data"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/","url":"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/","name":"Documentum - FT - FTIntegrity Tool - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2019-12-03T17:00:55+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/deb907c3360cacdc6c7df54b4bac3c86"},"breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/documentum-ft-ftintegrity-tool\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Documentum &#8211; FT &#8211; FTIntegrity Tool"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/deb907c3360cacdc6c7df54b4bac3c86","name":"David Diab","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/212b1b2e4650bad3116f644ab4fb4663786d94195d7685d0704c8426da088e60?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/212b1b2e4650bad3116f644ab4fb4663786d94195d7685d0704c8426da088e60?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/212b1b2e4650bad3116f644ab4fb4663786d94195d7685d0704c8426da088e60?s=96&d=mm&r=g","caption":"David Diab"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/david-diab\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/13137","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/46"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=13137"}],"version-history":[{"count":0,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/13137\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=13137"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=13137"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=13137"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=13137"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}