A couple year ago at a new customer, I was asked to install a new xPlore IndexAgent on an existing Documentum 16.4 environment. Unfortunately, that installation didn’t succeed, and I ended up with the following install logs:

[xplore@ds-0 ~]$ cd $XPLORE_HOME/setup/indexagent/logs
[xplore@ds-0 logs]$
[xplore@ds-0 logs]$ cat install.log
09:33:02,618  INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - The product name is: Index Agent
09:33:02,619  INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - The product version is: 16.4.0000.0197
09:33:02,619  INFO [main]  -
09:33:02,647  INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - Done InitializeSharedLibrary ...
09:33:11,929  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - setApplicationServer sharedDfcLibDir is:$XPLORE_HOME/dfc
09:33:11,930  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - getFileFromResource for templates/appserver.properties
09:33:12,006  INFO [main] com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions - starting DctmActions
09:33:14,381  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - user name =
09:33:14,382  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - The batch file: $XPLORE_HOME/temp/installer/wildfly/dctm_tmpcmd0.sh exist? false
09:33:14,383  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - The user home is : /home/xplore
09:33:14,386  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - Executing temporary batch file: $XPLORE_HOME/temp/installer/wildfly/dctm_tmpcmd0.sh for running: $JAVA_HOME/bin/java -cp $WILDFLY_HOME/modules/system/layers/base/emc/documentum/security/main/dfc.jar:$WILDFLY_HOME/modules/system/layers/base/emc/documentum/security/main/aspectjrt.jar:$WILDFLY_HOME/modules/system/layers/base/emc/documentum/security/main/DctmUtils.jar com.documentum.install.appserver.utils.DctmCreatePasswordHash xxxx
09:33:15,046  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - Deploying to Group Indexagent_REPO2... IndexAgent (IndexAgent.war): does not exist!
09:33:15,057  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/dfc.properties
09:33:15,065  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/log4j.properties
09:33:15,071  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/logback.xml
09:33:15,080  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/indexagent.xml
09:33:15,092  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/web.xml
09:33:15,102  INFO [main] com.documentum.install.shared.actions.DiActionCopyFilesTarget - copying $XPLORE_HOME/setup/indexagent/tools/templates/aclreplication.sh.template to $XPLORE_HOME/setup/indexagent/tools/aclreplication_for_REPO2.sh
09:33:15,120  INFO [main] com.documentum.install.shared.actions.DiActionCopyFilesTarget - copying $XPLORE_HOME/setup/indexagent/tools/templates/ftintegrity.sh.template to $XPLORE_HOME/setup/indexagent/tools/ftintegrity_for_REPO2.sh
09:33:15,130  INFO [main] com.documentum.install.shared.actions.DiActionSetPermissionTarget - performing chmod 755 $XPLORE_HOME/setup/indexagent/tools/aclreplication_for_REPO2.sh
09:33:15,137  INFO [main] com.documentum.install.shared.actions.DiActionSetPermissionTarget - performing chmod 755 $XPLORE_HOME/setup/indexagent/tools/ftintegrity_for_REPO2.sh
09:33:15,141  INFO [main] com.documentum.install.shared.actions.DiActionResolveMacrosTarget - performing macro resolution on $XPLORE_HOME/setup/indexagent/tools/aclreplication_for_REPO2.sh
09:33:15,153  INFO [main] com.documentum.install.shared.actions.DiActionResolveMacrosTarget - performing macro resolution on $XPLORE_HOME/setup/indexagent/tools/ftintegrity_for_REPO2.sh
09:33:15,166  INFO [main] com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions - Finished DctmActions.
09:33:15,571  INFO [main] com.documentum.fc.client.security.impl.JKSKeystoreUtilForDfc - keystore file name is $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/dfc.keystore
09:33:15,945 ERROR [main] com.documentum.fc.client.security.impl.IdentityManager - [DFC_SECURITY_IDENTITY_INIT] no identity initialization or incomplete identity initialization
com.documentum.fc.common.DfCriticalException:
        at com.documentum.fc.client.security.internal.CreateIdentityCredential$MultiFormatPKIKeyPair.<init>(CreateIdentityCredential.java:734)
        at com.documentum.fc.client.security.internal.CreateIdentityCredential.<init>(CreateIdentityCredential.java:98)
        at com.documentum.fc.client.security.impl.InitializeKeystoreForDfc.execute(InitializeKeystoreForDfc.java:46)
        at com.documentum.fc.client.security.internal.KeystoreMgr.initForDfcWithDefaultPolicy(KeystoreMgr.java:71)
        at com.documentum.fc.client.security.impl.IdentityManager.initFromKeystore(IdentityManager.java:139)
        at com.documentum.fc.client.security.impl.IdentityManager.<init>(IdentityManager.java:41)
        at com.documentum.fc.impl.RuntimeContext.<init>(RuntimeContext.java:52)
        at com.documentum.fc.impl.RuntimeContext.<clinit>(RuntimeContext.java:185)
        at com.documentum.fc.client.DfClient.<clinit>(DfClient.java:772)
        at com.documentum.com.DfClientX.getLocalClient(DfClientX.java:43)
        at com.documentum.server.impl.fulltext.indexagent.install.SessionController.createSessionManager(SessionController.java:43)
        at com.documentum.server.impl.fulltext.indexagent.install.SessionController.getSessionManager(SessionController.java:20)
        at com.documentum.server.impl.fulltext.indexagent.install.SessionController.getSession(SessionController.java:61)
        at com.documentum.server.impl.fulltext.indexagent.install.DocbaseSetup.<init>(DocbaseSetup.java:71)
        at com.documentum.install.indexagent.services.Docbase.<init>(Docbase.java:27)
        at com.documentum.install.indexagent.services.Docbase.<init>(Docbase.java:22)
        at com.documentum.install.indexagent.services.Docbase.createFulltextObjectsUsingMacros(Docbase.java:72)
        at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.configureFulltextDocbaseObjects(ConfigIndexAgentActions.java:367)
        at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.createIndexAgent(ConfigIndexAgentActions.java:107)
        at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.install(ConfigIndexAgentActions.java:50)
        at com.zerog.ia.installer.actions.CustomAction.installSelf(Unknown Source)
        at com.zerog.ia.installer.InstallablePiece.install(Unknown Source)
        at com.zerog.ia.installer.InstallablePiece.install(Unknown Source)
        at com.zerog.ia.installer.GhostDirectory.install(Unknown Source)
        at com.zerog.ia.installer.InstallablePiece.install(Unknown Source)
        at com.zerog.ia.installer.Installer.install(Unknown Source)
        at com.zerog.ia.installer.LifeCycleManager.consoleInstallMain(Unknown Source)
        at com.zerog.ia.installer.LifeCycleManager.executeApplication(Unknown Source)
        at com.zerog.ia.installer.Main.main(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.zerog.lax.LAX.launch(Unknown Source)
        at com.zerog.lax.LAX.main(Unknown Source)
Caused by: java.security.NoSuchAlgorithmException: A JSAFE_SecureRandom object of HMACDRBG256 is not available on any of the devices. (Java)
        at com.rsa.jsafe.JSAFE_SecureRandom.a(Unknown Source)
        at com.rsa.jsafe.JSAFE_SecureRandom.getInstance(Unknown Source)
        at com.documentum.fc.client.security.internal.CreateIdentityCredential$MultiFormatPKIKeyPair.<init>(CreateIdentityCredential.java:601)
        ... 34 more
09:33:15,948  INFO [main] com.documentum.fc.impl.RuntimeContext - DFC Version is 16.4.0000.0185
09:33:15,982  INFO [Timer-2] com.documentum.fc.client.impl.bof.cache.ClassCacheManager$CacheCleanupTask - [DFC_BOF_RUNNING_CLEANUP] Running class cache cleanup task
09:33:17,069  INFO [main] com.documentum.fc.client.impl.connection.docbase.DocbaseConnection - Object protocol version 2
09:33:17,087 ERROR [main] com.documentum.fc.client.security.impl.IdentityManager - [DFC_SECURITY_IDENTITY_CREATION] failure on creation of identity: 'bad bundle'
DfException:: THREAD: main; MSG: [DFC_SECURITY_IDENTITY_BUNDLE_FAIL] could not create identity bundle because identity initialization failed; ERRORCODE: ff; NEXT: null
        at com.documentum.fc.client.security.impl.IdentityManager.createIdentityBundle(IdentityManager.java:188)
        at com.documentum.fc.client.security.impl.IdentityManager.getIdentityBundle(IdentityManager.java:51)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.authenticate(DocbaseConnection.java:413)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.open(DocbaseConnection.java:131)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.<init>(DocbaseConnection.java:100)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.<init>(DocbaseConnection.java:60)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionFactory.newDocbaseConnection(DocbaseConnectionFactory.java:26)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.createNewConnection(DocbaseConnectionManager.java:192)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.getDocbaseConnection(DocbaseConnectionManager.java:122)
        at com.documentum.fc.client.impl.session.SessionFactory.newSession(SessionFactory.java:23)
        at com.documentum.fc.client.impl.session.PrincipalAwareSessionFactory.newSession(PrincipalAwareSessionFactory.java:44)
        at com.documentum.fc.client.impl.session.PooledSessionFactory.newSession(PooledSessionFactory.java:49)
        at com.documentum.fc.client.impl.session.SessionManager.getSessionFromFactory(SessionManager.java:134)
        at com.documentum.fc.client.impl.session.SessionManager.newSession(SessionManager.java:72)
        at com.documentum.fc.client.impl.session.SessionManager.getSession(SessionManager.java:191)
        at com.documentum.fc.client.impl.bof.classmgmt.ModuleManager.connect(ModuleManager.java:397)
        at com.documentum.fc.client.impl.bof.classmgmt.ModuleManager.init(ModuleManager.java:352)
        at com.documentum.fc.client.impl.bof.classmgmt.ModuleManager.getInstance(ModuleManager.java:43)
        at com.documentum.fc.client.security.impl.DfcIdentityPublisher.<init>(DfcIdentityPublisher.java:44)
        at com.documentum.fc.client.security.internal.RegistrationMgr.register(RegistrationMgr.java:34)
        at com.documentum.fc.impl.RuntimeContext.<clinit>(RuntimeContext.java:195)
        at com.documentum.fc.client.DfClient.<clinit>(DfClient.java:772)
        at com.documentum.com.DfClientX.getLocalClient(DfClientX.java:43)
        at com.documentum.server.impl.fulltext.indexagent.install.SessionController.createSessionManager(SessionController.java:43)
        at com.documentum.server.impl.fulltext.indexagent.install.SessionController.getSessionManager(SessionController.java:20)
        at com.documentum.server.impl.fulltext.indexagent.install.SessionController.getSession(SessionController.java:61)
        at com.documentum.server.impl.fulltext.indexagent.install.DocbaseSetup.<init>(DocbaseSetup.java:71)
        at com.documentum.install.indexagent.services.Docbase.<init>(Docbase.java:27)
        at com.documentum.install.indexagent.services.Docbase.<init>(Docbase.java:22)
        at com.documentum.install.indexagent.services.Docbase.createFulltextObjectsUsingMacros(Docbase.java:72)
        at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.configureFulltextDocbaseObjects(ConfigIndexAgentActions.java:367)
        at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.createIndexAgent(ConfigIndexAgentActions.java:107)
        at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.install(ConfigIndexAgentActions.java:50)
        at com.zerog.ia.installer.actions.CustomAction.installSelf(Unknown Source)
        at com.zerog.ia.installer.InstallablePiece.install(Unknown Source)
        at com.zerog.ia.installer.InstallablePiece.install(Unknown Source)
        at com.zerog.ia.installer.GhostDirectory.install(Unknown Source)
        at com.zerog.ia.installer.InstallablePiece.install(Unknown Source)
        at com.zerog.ia.installer.Installer.install(Unknown Source)
        at com.zerog.ia.installer.LifeCycleManager.consoleInstallMain(Unknown Source)
        at com.zerog.ia.installer.LifeCycleManager.executeApplication(Unknown Source)
        at com.zerog.ia.installer.Main.main(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.zerog.lax.LAX.launch(Unknown Source)
        at com.zerog.lax.LAX.main(Unknown Source)
09:33:17,373  INFO [main] com.documentum.fc.client.security.impl.JKSKeystoreUtilForDfc - keystore file name is $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/dfc.keystore
09:33:17,395 ERROR [main] com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions - Failed to configure fulltext repository objects.
java.lang.Exception: java.lang.ExceptionInInitializerError
        at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.configureFulltextDocbaseObjects(ConfigIndexAgentActions.java:372)
        at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.createIndexAgent(ConfigIndexAgentActions.java:107)
        at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.install(ConfigIndexAgentActions.java:50)
        at com.zerog.ia.installer.actions.CustomAction.installSelf(Unknown Source)
        at com.zerog.ia.installer.InstallablePiece.install(Unknown Source)
        at com.zerog.ia.installer.InstallablePiece.install(Unknown Source)
        at com.zerog.ia.installer.GhostDirectory.install(Unknown Source)
        at com.zerog.ia.installer.InstallablePiece.install(Unknown Source)
        at com.zerog.ia.installer.Installer.install(Unknown Source)
        at com.zerog.ia.installer.LifeCycleManager.consoleInstallMain(Unknown Source)
        at com.zerog.ia.installer.LifeCycleManager.executeApplication(Unknown Source)
        at com.zerog.ia.installer.Main.main(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.zerog.lax.LAX.launch(Unknown Source)
        at com.zerog.lax.LAX.main(Unknown Source)
Caused by: java.lang.ExceptionInInitializerError
        at com.documentum.fc.client.DfClient.<clinit>(DfClient.java:772)
        at com.documentum.com.DfClientX.getLocalClient(DfClientX.java:43)
        at com.documentum.server.impl.fulltext.indexagent.install.SessionController.createSessionManager(SessionController.java:43)
        at com.documentum.server.impl.fulltext.indexagent.install.SessionController.getSessionManager(SessionController.java:20)
        at com.documentum.server.impl.fulltext.indexagent.install.SessionController.getSession(SessionController.java:61)
        at com.documentum.server.impl.fulltext.indexagent.install.DocbaseSetup.<init>(DocbaseSetup.java:71)
        at com.documentum.install.indexagent.services.Docbase.<init>(Docbase.java:27)
        at com.documentum.install.indexagent.services.Docbase.<init>(Docbase.java:22)
        at com.documentum.install.indexagent.services.Docbase.createFulltextObjectsUsingMacros(Docbase.java:72)
        at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.configureFulltextDocbaseObjects(ConfigIndexAgentActions.java:367)
        ... 17 more
Caused by: java.lang.NullPointerException
        at com.documentum.fc.client.security.impl.IpAndRcHelper.getRegistration(IpAndRcHelper.java:439)
        at com.documentum.fc.client.security.impl.DfcIdentityPublisher.getRegistration(DfcIdentityPublisher.java:475)
        at com.documentum.fc.client.security.impl.DfcIdentityPublisher.publish(DfcIdentityPublisher.java:134)
        at com.documentum.fc.client.security.impl.DfcIdentityPublisher.publish(DfcIdentityPublisher.java:102)
        at com.documentum.fc.client.security.internal.RegistrationMgr.register(RegistrationMgr.java:34)
        at com.documentum.fc.impl.RuntimeContext.<clinit>(RuntimeContext.java:195)
        ... 27 more
09:33:17,396  INFO [main]  - The INSTALLER_UI value is SILENT
09:33:17,396  INFO [main]  - The common.installLocation value is $XPLORE_HOME
09:33:17,396  INFO [main]  - The indexagent.configMode.create value is 1
09:33:17,396  INFO [main]  - The indexagent.configMode.upgrade value is 0
09:33:17,396  INFO [main]  - The indexagent.configMode.delete value is 0
09:33:17,396  INFO [main]  - The indexagent.configMode.create.migration value is 0
09:33:17,397  INFO [main]  - The indexagent.ess.host value is ds-0.ds.namespace.svc.cluster.local
09:33:17,397  INFO [main]  - The indexagent.ess.port value is 9300
09:33:17,397  INFO [main]  - The indexagent.name value is Indexagent_REPO2
09:33:17,397  INFO [main]  - The indexagent.FQDN value is ds-0.ds.namespace.svc.cluster.local
09:33:17,397  INFO [main]  - The indexagent.instance.port value is 9220
09:33:17,397  INFO [main]  - The indexagent.instance.password value is ******
09:33:17,397  INFO [main]  - The indexagent.docbase.name value is REPO2
09:33:17,397  INFO [main]  - The indexagent.docbase.user value is dmadmin
09:33:17,397  INFO [main]  - The indexagent.docbase.password value is ******
09:33:17,398  INFO [main]  - The indexagent.connectionBroker.host value is cs.namespace.svc.cluster.local
09:33:17,398  INFO [main]  - The indexagent.connectionBroker.port value is 1489
09:33:17,398  INFO [main]  - The indexagent.globalRegistryRepository.name value is gr_REPO
09:33:17,398  INFO [main]  - The indexagent.globalRegistryRepository.user value is dm_bof_registry
09:33:17,398  INFO [main]  - The indexagent.globalRegistryRepository.password value is ******
09:33:17,398  INFO [main]  - The indexagent.storage.name value is default
09:33:17,398  INFO [main]  - The indexagent.local_content_area value is /data/ds-0/data/export/ia/Indexagent_REPO2
09:33:17,398  INFO [main]  - The common.installOwner.password value is ******
09:33:17,398  INFO [main]  - The indexagent.delete.objects value is 1
09:33:17,398  INFO [main]  - The indexagent.STANDBY_FULLTEXT_USER value is dm_fulltext_index_user_02
09:33:17,398  INFO [main]  - The indexagent.STANDBY_FULLTEXT_INDEX_NAME value is REPO2_ftindex_02
09:33:17,399  INFO [main]  - The indexagent.STANDBY_FULLTEXT_ENGINE_NAME value is DSearch Fulltext Engine Configuration 02
09:33:17,399  INFO [main]  - The env PATH value is: /usr/xpg4/bin:$JAVA_HOME/bin:$JAVA_HOME/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/xplore/.local/bin:/home/xplore/bin:/sbin
09:33:17,445  WARN [main] com.documentum.install.indexagent.installanywhere.actions.RegisterIndexAgentWithWatchdog - Stopping watchdog.
09:33:17,445  WARN [main] com.documentum.install.indexagent.installanywhere.actions.RegisterIndexAgentWithWatchdog - Registering ds-0.ds.namespa_9220_IndexAgent with watchdog.
09:33:17,793  INFO [main] com.emc.ess.watchdog.config.registry.impl.EssMultiWatchdogConfiguration - Begin to add SingleWatchdogConfig with hostname: ds-0.ds.namespace.svc.cluster.local
09:33:17,795  INFO [main] com.emc.ess.watchdog.config.EssWatchdogRegistration - Successfully createdcom.emc.ess.watchdog.common.impl.tasks.SendFailureNotificationTask
09:33:17,795  INFO [main] com.emc.ess.watchdog.config.EssWatchdogRegistration - Successfully created IndexAgent task ds-0.ds.namespa_9220_IndexAgent_HeartBeat for application instance ds-0.ds.namespa_9220_IndexAgent
09:33:17,888  INFO [main] com.emc.ess.watchdog.config.EssWatchdogRegistration - Successfully added configuration for  {IndexAgent,ds-0.ds.namespa_9220_IndexAgent} under watchdog host ds-0.ds.namespace.svc.cluster.local
09:33:17,888  WARN [main] com.documentum.install.indexagent.installanywhere.actions.RegisterIndexAgentWithWatchdog - Registration succeeded.
[xplore@ds-0 logs]$

As you probably know, xPlore 16.4 is now quite old and it was still a version that came out-of-the-box with the Oracle JDK 8. The issue above is that the DFC Keystore couldn’t be initialized/created for this new IndexAgent and the reason for that is shown in the log file as “NoSuchAlgorithmException: A JSAFE_SecureRandom object of HMACDRBG256 is not available“. HMACDRBG256 is the default Pseudo-Random Number Generator (PRNG) algorithm for FIPS, so that made me think about the cryptoj and other jars that will be needed for RSA/FIPS.

Therefore, the first thing I did was check the current Java version as well as the status of the RSA/FIPS libraries:

[xplore@ds-0 logs]$ cd $JAVA_HOME/jre/lib/ext
[xplore@ds-0 ext]$
[xplore@ds-0 ext]$ java -version
java version "1.8.0_261"
Java(TM) SE Runtime Environment (build 1.8.0_261-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.261-b12, mixed mode)
[xplore@ds-0 ext]$
[xplore@ds-0 ext]$ ls -ltr
total 27792
-rw-r----- 1 xplore xplore 18493706 Jun 18  2020 jfxrt.jar
-rw-r----- 1 xplore xplore    69304 Jun 18  2020 zipfs.jar
-rw-r----- 1 xplore xplore   280139 Jun 18  2020 sunpkcs11.jar
-rw-r----- 1 xplore xplore   280999 Jun 18  2020 sunjce_provider.jar
-rw-r----- 1 xplore xplore    60361 Jun 18  2020 sunec.jar
-rw-r----- 1 xplore xplore  2024157 Jun 18  2020 nashorn.jar
-rw-r----- 1 xplore xplore  1179462 Jun 18  2020 localedata.jar
-rw-r----- 1 xplore xplore    44516 Jun 18  2020 jaccess.jar
-rw-r----- 1 xplore xplore     8286 Jun 18  2020 dnsns.jar
-rw-r----- 1 xplore xplore  3860563 Jun 18  2020 cldrdata.jar
-rw-r----- 1 xplore xplore     1269 Jun 18  2020 meta-index
[xplore@ds-0 ext]$

As you can see, it looked like someone updated/patched the Oracle JDK 8 (I believe the default that comes with xPlore 16.4 being 1.8.0_152 and not 1.8.0_261) but while doing so, only the base JARs were put in place and not the RSA/FIPS related ones. So, it looks like something was missed in the patching steps. These additional JARs aren’t really needed for the runtime, but they are when trying to install a new IndexAgent because of the DFC Keystore creation, so that probably explained why the customer didn’t see any issues in the few months after the JDK upgrade.

I quickly installed an OOTB xPlore 16.4 environment (with the OOTB JDK 8u152) to be able to list all the “custom” JARs present so that I could take them and restore these into the environment with patched JDK. This is the list of JARs (OOTB + “custom”):

[xplore@ds-0 ext]$ ls -ltr
total 29700
-rw-r----- 1 xplore xplore 18513457 Sep 14  2017 jfxrt.jar
-rw-r----- 1 xplore xplore    68965 Sep 14  2017 zipfs.jar
-rw-r----- 1 xplore xplore   248726 Sep 14  2017 sunpkcs11.jar
-rw-r----- 1 xplore xplore   273952 Sep 14  2017 sunjce_provider.jar
-rw-r----- 1 xplore xplore    41672 Sep 14  2017 sunec.jar
-rw-r----- 1 xplore xplore  2023991 Sep 14  2017 nashorn.jar
-rw-r----- 1 xplore xplore      746 Sep 14  2017 meta-index
-rw-r----- 1 xplore xplore  1179101 Sep 14  2017 localedata.jar
-rw-r----- 1 xplore xplore    44516 Sep 14  2017 jaccess.jar
-rw-r----- 1 xplore xplore     8286 Sep 14  2017 dnsns.jar
-rw-r----- 1 xplore xplore  3860502 Sep 14  2017 cldrdata.jar
-rw-r----- 1 xplore xplore     8046 Oct 15 09:41 util.jar
-rw-r----- 1 xplore xplore   421576 Oct 15 09:41 sslj.jar
-rw-r----- 1 xplore xplore   505314 Oct 15 09:41 jcm.jar
-rw-r----- 1 xplore xplore   506533 Oct 15 09:41 jcmFIPS.jar
-rw-r----- 1 xplore xplore   325610 Oct 15 09:41 jcmandroidfips.jar
-rw-r----- 1 xplore xplore  1548393 Oct 15 09:41 cryptojcommon.jar
-rw-r----- 1 xplore xplore    70096 Oct 15 09:41 cryptojce.jar
-rw-r----- 1 xplore xplore   727472 Oct 15 09:41 certj.jar
[xplore@ds-0 ext]$

As you can see above, there are 11 OOTB JARs inside a Java 8u152 and then xPlore during its installation will deploy an additional 8 JARs for RSA/FIPS. Therefore, it looks like the person that upgraded the JDK forgot to restore these 8 JARs into the new JDK folder. For information, this is documented in the xPlore Patch Notes under the “Upgrading Bundled JDK 8” section.

I restored the 8 “custom” JARs related to RSA/FIPS into the upgraded JDK 8u261 and then started again the xPlore IndexAgent installation, which succeeded this time:

[xplore@ds-0 ~]$ cd $JAVA_HOME/jre/lib/ext
[xplore@ds-0 ext]$
[xplore@ds-0 ext]$ ls -ltr
total 29740
-rw-r----- 1 xplore xplore 18493706 Jun 18  2020 jfxrt.jar
-rw-r----- 1 xplore xplore    69304 Jun 18  2020 zipfs.jar
-rw-r----- 1 xplore xplore   280139 Jun 18  2020 sunpkcs11.jar
-rw-r----- 1 xplore xplore   280999 Jun 18  2020 sunjce_provider.jar
-rw-r----- 1 xplore xplore    60361 Jun 18  2020 sunec.jar
-rw-r----- 1 xplore xplore  2024157 Jun 18  2020 nashorn.jar
-rw-r----- 1 xplore xplore  1179462 Jun 18  2020 localedata.jar
-rw-r----- 1 xplore xplore    44516 Jun 18  2020 jaccess.jar
-rw-r----- 1 xplore xplore     8286 Jun 18  2020 dnsns.jar
-rw-r----- 1 xplore xplore  3860563 Jun 18  2020 cldrdata.jar
-rw-r----- 1 xplore xplore     1269 Jun 18  2020 meta-index
-rw-r----- 1 xplore xplore     8046 Oct 15 10:20 util.jar
-rw-r----- 1 xplore xplore   421576 Oct 15 10:20 sslj.jar
-rw-r----- 1 xplore xplore   505314 Oct 15 10:20 jcm.jar
-rw-r----- 1 xplore xplore   506533 Oct 15 10:20 jcmFIPS.jar
-rw-r----- 1 xplore xplore   325610 Oct 15 10:20 jcmandroidfips.jar
-rw-r----- 1 xplore xplore  1548393 Oct 15 10:20 cryptojcommon.jar
-rw-r----- 1 xplore xplore    70096 Oct 15 10:20 cryptojce.jar
-rw-r----- 1 xplore xplore   727472 Oct 15 10:20 certj.jar
[xplore@ds-0 ext]$

[xplore@ds-0 ext]$ cd $XPLORE_HOME/setup/indexagent/logs
[xplore@ds-0 logs]$
[xplore@ds-0 logs]$ cat install.log
10:50:18,085  INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - The product name is: Index Agent
10:50:18,085  INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - The product version is: 16.4.0000.0197
10:50:18,085  INFO [main]  -
10:50:18,126  INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - Done InitializeSharedLibrary ...
10:50:27,513  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - setApplicationServer sharedDfcLibDir is:$XPLORE_HOME/dfc
10:50:27,514  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - getFileFromResource for templates/appserver.properties
10:50:27,596  INFO [main] com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions - starting DctmActions
10:50:32,428  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - user name =
10:50:32,430  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - The batch file: $XPLORE_HOME/temp/installer/wildfly/dctm_tmpcmd0.sh exist? false
10:50:32,430  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - The user home is : /home/xplore
10:50:32,435  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - Executing temporary batch file: $XPLORE_HOME/temp/installer/wildfly/dctm_tmpcmd0.sh for running: $JAVA_HOME/bin/java -cp $WILDFLY_HOME/modules/system/layers/base/emc/documentum/security/main/dfc.jar:$WILDFLY_HOME/modules/system/layers/base/emc/documentum/security/main/aspectjrt.jar:$WILDFLY_HOME/modules/system/layers/base/emc/documentum/security/main/DctmUtils.jar com.documentum.install.appserver.utils.DctmCreatePasswordHash xxxx
10:50:32,867  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - Server DctmServer_Indexagent_REPO2 already exists!
10:50:32,868  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - Deploying to Group Indexagent_REPO2... IndexAgent (IndexAgent.war): does not exist!
10:50:32,872  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/dfc.properties
10:50:32,879  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/log4j.properties
10:50:32,886  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/logback.xml
10:50:32,893  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/indexagent.xml
10:50:32,904  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/web.xml
10:50:32,914  INFO [main] com.documentum.install.shared.actions.DiActionCopyFilesTarget - copying $XPLORE_HOME/setup/indexagent/tools/templates/aclreplication.sh.template to $XPLORE_HOME/setup/indexagent/tools/aclreplication_for_REPO2.sh
10:50:32,931  INFO [main] com.documentum.install.shared.actions.DiActionCopyFilesTarget - copying $XPLORE_HOME/setup/indexagent/tools/templates/ftintegrity.sh.template to $XPLORE_HOME/setup/indexagent/tools/ftintegrity_for_REPO2.sh
10:50:32,949  INFO [main] com.documentum.install.shared.actions.DiActionSetPermissionTarget - performing chmod 755 $XPLORE_HOME/setup/indexagent/tools/aclreplication_for_REPO2.sh
10:50:32,955  INFO [main] com.documentum.install.shared.actions.DiActionSetPermissionTarget - performing chmod 755 $XPLORE_HOME/setup/indexagent/tools/ftintegrity_for_REPO2.sh
10:50:32,962  INFO [main] com.documentum.install.shared.actions.DiActionResolveMacrosTarget - performing macro resolution on $XPLORE_HOME/setup/indexagent/tools/aclreplication_for_REPO2.sh
10:50:32,978  INFO [main] com.documentum.install.shared.actions.DiActionResolveMacrosTarget - performing macro resolution on $XPLORE_HOME/setup/indexagent/tools/ftintegrity_for_REPO2.sh
10:50:32,989  INFO [main] com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions - Finished DctmActions.
10:50:33,385  INFO [main] com.documentum.fc.client.security.impl.JKSKeystoreUtilForDfc - keystore file name is $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/dfc.keystore
10:50:34,002  INFO [main] com.documentum.fc.client.security.internal.CreateIdentityCredential$MultiFormatPKIKeyPair - generated RSA (2,048-bit strength) mutiformat key pair in 328 ms
10:50:34,028  INFO [main] com.documentum.fc.client.security.internal.CreateIdentityCredential - certificate created for DFC <CN=dfc_ZpcFYVkXHfBSo51C2rp7HXSbxbwa,O=EMC,OU=Documentum> valid from Fri Oct 15 10:45:33 UTC 2021 to Mon Oct 13 10:50:33 UTC 2031:

10:50:34,032  INFO [main] com.documentum.fc.client.security.impl.JKSKeystoreUtilForDfc - keystore file name is $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/dfc.keystore
10:50:34,055  INFO [main] com.documentum.fc.client.security.impl.InitializeKeystoreForDfc - [DFC_SECURITY_IDENTITY_INITIALIZED] Initialized new identity in keystore, DFC alias=dfc, identity=dfc_ZpcFYVkXHfBSo51C2rp7HXSbxbwa
10:50:34,056  INFO [main] com.documentum.fc.client.security.impl.AuthenticationMgrForDfc - identity for authentication is dfc_ZpcFYVkXHfBSo51C2rp7HXSbxbwa
10:50:34,057  INFO [main] com.documentum.fc.impl.RuntimeContext - DFC Version is 16.4.0000.0185
10:50:34,078  INFO [Timer-2] com.documentum.fc.client.impl.bof.cache.ClassCacheManager$CacheCleanupTask - [DFC_BOF_RUNNING_CLEANUP] Running class cache cleanup task
10:50:35,346  INFO [main] com.documentum.fc.client.impl.connection.docbase.DocbaseConnection - Object protocol version 2
10:50:35,402  INFO [main] com.documentum.fc.client.security.internal.AuthenticationMgr - new identity bundle <dfc_ZpcFYVkXHfBSo51C2rp7HXSbxbwa  1634295035      ds-0.ds.namespace.svc.cluster.local           YnmJpnz48aXW8QuEkgQm/eHbq9hCgJnUGCxdjmPsUJzEnEr5nSwk3KPQ2WUo5ihIuylu2ihLt5o8mTpsEoIBt177lMnAvtU+nbBtB0ItTnD4Bt1M83IQR+t5JAQSYF7Yk7WYCiAADEcKWno6/JiEgkYM0qjU5HwR0w/zmxDSf2330KImqYRPLQtEdCoW8Z3+SSs2/xJdrcki27of9CKH5BXBaXpGpqFzBCYYy76C+SL5+GUR8srU2Q9d4qo5GrlaLJecyWw1Oy3ZZbxDX/nzn7PmBOemkS8rqHZK5msTW9Z8G7Lt6WfXv6w3qumAYJhJCXnXtQEd0TG2liM5XWGqVg==>
10:50:35,689  INFO [main] com.documentum.fc.client.security.impl.JKSKeystoreUtilForDfc - keystore file name is $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/dfc.keystore
10:50:35,734  INFO [main] com.documentum.fc.client.security.impl.DfcIdentityPublisher - found client registration: false
10:50:38,098  INFO [main] com.documentum.fc.client.privilege.impl.PublicKeyCertificate - stored certificate for CN
10:50:38,262  INFO [main] com.documentum.fc.client.security.impl.IpAndRcHelper - filling in gr_REPO a new record with this persistent certificate:
-----BEGIN CERTIFICATE-----
MIIDHzCCAgcCEBVLERfLzAbZ/gi5w7PVnM4wDQYJKoZIhvcNAQELBQAwTjETMBEG
A1UECwwKRG9jdW1lbnR1bTEMMAoGA1UECgwDRU1DMSkwJwYDVQQDDCBkZmNfWlhT
YnhrWHBjRllWMnJwN0hIZkJTbzUxQ2J3YTAeFw0yMTEwMTUxMDQ1MzNaFw0zMTEw
tvbnbJm2IAOE1a1w/dK+NTltmB4QvOoVAvOkaiuTN6qMmi4zCQRjDssaxfdnn9xV
K9lj+v+lXdjlhkCg8PGIssjaAd+zxiVPK9+so52pt2U9rwSk/ASXZqpa3sERO0ua
CJtlPyHuCXHVB6yrwwgnkyWO0PFyfWTi4DUMKJpQCjXGR17+EkfS2u5dGtEyeiLt
CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCiBaGp0Wh0hp3/xptFspRkPKaDoOB3
phrVNQqiJT3CiqJgFabiIEGS3AOkm85372RaS4ZrqUh++P5LEkflFZuRzvgmeLMj
HAH5VPSKM8QE3VrVQo17X1msfq4HeRXpUYPk7iL9HPoDSrL7J+kphTPrAgMBAAEw
DQYJKoZIhvcNAQELBQADggEBAB4pqKL0J7C63/IyJGXN/jZXaN8Fg2bz5bYythgI
MTMxMDUwMzNaME4xEzARBgNVBAsMCkRvY3VtZW50dW0xDDAKBgNVBAoMA0VNQzEp
MCcGA1UEAwwgZGZjX1pYU2J4a1hwY0ZZVjJycDdISGZCU281MUNid2EwggEiMA0G
ZwF0p0Io6oDVXjK921c7TovGz8SdxVARSR/LG7EAXYAqkvXkXMkow65BonvBPZcE
Y6oYUq7OrtWTWd+yOryZ+y+TwG0P2cDd5nErCSoiuNlJ+MvhvJ/fraknJsr9/iGD
rZvZpAT0aRAKwKaj65RlK6Pj43B6rPlouD1yS6Kgb5KpgOCuK/HxpcseUsmNDh5v
JRD05r0VxD6b3nMqRP3E7XPr3hbEbq79R6qF8aaF+mm4SZNSEgm1Ps2ruzchwShe
Xn+DBv7XA0xCOmHbLvLe/tLlNKH8GPCuqcbcSUTTCmyE8/A=
-----END CERTIFICATE-----
10:50:38,266  INFO [main] com.documentum.fc.client.security.impl.DfcIdentityPublisher - found client registration: false
10:50:38,454  INFO [main] com.documentum.fc.client.security.impl.IpAndRcHelper - filling a new registration record for dfc_ZpcFYVkXHfBSo51C2rp7HXSbxbwa
10:50:38,478  INFO [main] com.documentum.fc.client.security.impl.DfcIdentityPublisher - [DFC_SECURITY_GR_REGISTRATION_PUBLISH] this dfc instance is now published in the global registry gr_REPO
10:50:39,542  INFO [main] com.documentum.fc.client.impl.connection.docbase.DocbaseConnection - Object protocol version 2
10:50:45,788  INFO [main] com.documentum.fc.client.impl.connection.docbase.DocbaseConnection - Object protocol version 2
10:50:46,549  INFO [main] com.documentum.fc.client.impl.connection.docbase.DocbaseConnection - Object protocol version 2
10:50:47,848  INFO [main] com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions - storage name is set to: default
10:50:49,292  INFO [main] com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions - creating $XPLORE_HOME/installinfo/instances/indexagent
10:50:49,332  WARN [main] com.documentum.install.indexagent.installanywhere.actions.RegisterIndexAgentWithWatchdog - Stopping watchdog.
10:50:49,332  WARN [main] com.documentum.install.indexagent.installanywhere.actions.RegisterIndexAgentWithWatchdog - Registering ds-0.ds.namespa_9220_IndexAgent with watchdog.
10:50:49,568  INFO [main] com.emc.ess.watchdog.config.registry.impl.EssMultiWatchdogConfiguration - Begin to add SingleWatchdogConfig with hostname: ds-0.ds.namespace.svc.cluster.local
10:50:49,568  INFO [main] com.emc.ess.watchdog.config.EssWatchdogRegistration - Successfully added configuration for  {IndexAgent,ds-0.ds.namespa_9220_IndexAgent} under watchdog host ds-0.ds.namespace.svc.cluster.local
10:50:49,568  WARN [main] com.documentum.install.indexagent.installanywhere.actions.RegisterIndexAgentWithWatchdog - Registration succeeded.
[xplore@ds-0 logs]$

Long story short, whenever you want to try to patch something, always check the documentation to make sure you aren’t forgetting something! Even if the system can start and run properly after the patching, you might cause some issues to other people in the future like it was the case here ;).

Post Views: 598
Thumbnail [60x60]
by
Morgan Patou