A couple year ago at a new customer, I was asked to install a new xPlore IndexAgent on an existing Documentum 16.4 environment. Unfortunately, that installation didn’t succeed, and I ended up with the following install logs:
[xplore@ds-0 ~]$ cd $XPLORE_HOME/setup/indexagent/logs
[xplore@ds-0 logs]$
[xplore@ds-0 logs]$ cat install.log
09:33:02,618 INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - The product name is: Index Agent
09:33:02,619 INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - The product version is: 16.4.0000.0197
09:33:02,619 INFO [main] -
09:33:02,647 INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - Done InitializeSharedLibrary ...
09:33:11,929 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - setApplicationServer sharedDfcLibDir is:$XPLORE_HOME/dfc
09:33:11,930 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - getFileFromResource for templates/appserver.properties
09:33:12,006 INFO [main] com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions - starting DctmActions
09:33:14,381 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - user name =
09:33:14,382 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - The batch file: $XPLORE_HOME/temp/installer/wildfly/dctm_tmpcmd0.sh exist? false
09:33:14,383 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - The user home is : /home/xplore
09:33:14,386 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - Executing temporary batch file: $XPLORE_HOME/temp/installer/wildfly/dctm_tmpcmd0.sh for running: $JAVA_HOME/bin/java -cp $WILDFLY_HOME/modules/system/layers/base/emc/documentum/security/main/dfc.jar:$WILDFLY_HOME/modules/system/layers/base/emc/documentum/security/main/aspectjrt.jar:$WILDFLY_HOME/modules/system/layers/base/emc/documentum/security/main/DctmUtils.jar com.documentum.install.appserver.utils.DctmCreatePasswordHash xxxx
09:33:15,046 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - Deploying to Group Indexagent_REPO2... IndexAgent (IndexAgent.war): does not exist!
09:33:15,057 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/dfc.properties
09:33:15,065 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/log4j.properties
09:33:15,071 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/logback.xml
09:33:15,080 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/indexagent.xml
09:33:15,092 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/web.xml
09:33:15,102 INFO [main] com.documentum.install.shared.actions.DiActionCopyFilesTarget - copying $XPLORE_HOME/setup/indexagent/tools/templates/aclreplication.sh.template to $XPLORE_HOME/setup/indexagent/tools/aclreplication_for_REPO2.sh
09:33:15,120 INFO [main] com.documentum.install.shared.actions.DiActionCopyFilesTarget - copying $XPLORE_HOME/setup/indexagent/tools/templates/ftintegrity.sh.template to $XPLORE_HOME/setup/indexagent/tools/ftintegrity_for_REPO2.sh
09:33:15,130 INFO [main] com.documentum.install.shared.actions.DiActionSetPermissionTarget - performing chmod 755 $XPLORE_HOME/setup/indexagent/tools/aclreplication_for_REPO2.sh
09:33:15,137 INFO [main] com.documentum.install.shared.actions.DiActionSetPermissionTarget - performing chmod 755 $XPLORE_HOME/setup/indexagent/tools/ftintegrity_for_REPO2.sh
09:33:15,141 INFO [main] com.documentum.install.shared.actions.DiActionResolveMacrosTarget - performing macro resolution on $XPLORE_HOME/setup/indexagent/tools/aclreplication_for_REPO2.sh
09:33:15,153 INFO [main] com.documentum.install.shared.actions.DiActionResolveMacrosTarget - performing macro resolution on $XPLORE_HOME/setup/indexagent/tools/ftintegrity_for_REPO2.sh
09:33:15,166 INFO [main] com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions - Finished DctmActions.
09:33:15,571 INFO [main] com.documentum.fc.client.security.impl.JKSKeystoreUtilForDfc - keystore file name is $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/dfc.keystore
09:33:15,945 ERROR [main] com.documentum.fc.client.security.impl.IdentityManager - [DFC_SECURITY_IDENTITY_INIT] no identity initialization or incomplete identity initialization
com.documentum.fc.common.DfCriticalException:
at com.documentum.fc.client.security.internal.CreateIdentityCredential$MultiFormatPKIKeyPair.<init>(CreateIdentityCredential.java:734)
at com.documentum.fc.client.security.internal.CreateIdentityCredential.<init>(CreateIdentityCredential.java:98)
at com.documentum.fc.client.security.impl.InitializeKeystoreForDfc.execute(InitializeKeystoreForDfc.java:46)
at com.documentum.fc.client.security.internal.KeystoreMgr.initForDfcWithDefaultPolicy(KeystoreMgr.java:71)
at com.documentum.fc.client.security.impl.IdentityManager.initFromKeystore(IdentityManager.java:139)
at com.documentum.fc.client.security.impl.IdentityManager.<init>(IdentityManager.java:41)
at com.documentum.fc.impl.RuntimeContext.<init>(RuntimeContext.java:52)
at com.documentum.fc.impl.RuntimeContext.<clinit>(RuntimeContext.java:185)
at com.documentum.fc.client.DfClient.<clinit>(DfClient.java:772)
at com.documentum.com.DfClientX.getLocalClient(DfClientX.java:43)
at com.documentum.server.impl.fulltext.indexagent.install.SessionController.createSessionManager(SessionController.java:43)
at com.documentum.server.impl.fulltext.indexagent.install.SessionController.getSessionManager(SessionController.java:20)
at com.documentum.server.impl.fulltext.indexagent.install.SessionController.getSession(SessionController.java:61)
at com.documentum.server.impl.fulltext.indexagent.install.DocbaseSetup.<init>(DocbaseSetup.java:71)
at com.documentum.install.indexagent.services.Docbase.<init>(Docbase.java:27)
at com.documentum.install.indexagent.services.Docbase.<init>(Docbase.java:22)
at com.documentum.install.indexagent.services.Docbase.createFulltextObjectsUsingMacros(Docbase.java:72)
at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.configureFulltextDocbaseObjects(ConfigIndexAgentActions.java:367)
at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.createIndexAgent(ConfigIndexAgentActions.java:107)
at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.install(ConfigIndexAgentActions.java:50)
at com.zerog.ia.installer.actions.CustomAction.installSelf(Unknown Source)
at com.zerog.ia.installer.InstallablePiece.install(Unknown Source)
at com.zerog.ia.installer.InstallablePiece.install(Unknown Source)
at com.zerog.ia.installer.GhostDirectory.install(Unknown Source)
at com.zerog.ia.installer.InstallablePiece.install(Unknown Source)
at com.zerog.ia.installer.Installer.install(Unknown Source)
at com.zerog.ia.installer.LifeCycleManager.consoleInstallMain(Unknown Source)
at com.zerog.ia.installer.LifeCycleManager.executeApplication(Unknown Source)
at com.zerog.ia.installer.Main.main(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.zerog.lax.LAX.launch(Unknown Source)
at com.zerog.lax.LAX.main(Unknown Source)
Caused by: java.security.NoSuchAlgorithmException: A JSAFE_SecureRandom object of HMACDRBG256 is not available on any of the devices. (Java)
at com.rsa.jsafe.JSAFE_SecureRandom.a(Unknown Source)
at com.rsa.jsafe.JSAFE_SecureRandom.getInstance(Unknown Source)
at com.documentum.fc.client.security.internal.CreateIdentityCredential$MultiFormatPKIKeyPair.<init>(CreateIdentityCredential.java:601)
... 34 more
09:33:15,948 INFO [main] com.documentum.fc.impl.RuntimeContext - DFC Version is 16.4.0000.0185
09:33:15,982 INFO [Timer-2] com.documentum.fc.client.impl.bof.cache.ClassCacheManager$CacheCleanupTask - [DFC_BOF_RUNNING_CLEANUP] Running class cache cleanup task
09:33:17,069 INFO [main] com.documentum.fc.client.impl.connection.docbase.DocbaseConnection - Object protocol version 2
09:33:17,087 ERROR [main] com.documentum.fc.client.security.impl.IdentityManager - [DFC_SECURITY_IDENTITY_CREATION] failure on creation of identity: 'bad bundle'
DfException:: THREAD: main; MSG: [DFC_SECURITY_IDENTITY_BUNDLE_FAIL] could not create identity bundle because identity initialization failed; ERRORCODE: ff; NEXT: null
at com.documentum.fc.client.security.impl.IdentityManager.createIdentityBundle(IdentityManager.java:188)
at com.documentum.fc.client.security.impl.IdentityManager.getIdentityBundle(IdentityManager.java:51)
at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.authenticate(DocbaseConnection.java:413)
at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.open(DocbaseConnection.java:131)
at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.<init>(DocbaseConnection.java:100)
at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.<init>(DocbaseConnection.java:60)
at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionFactory.newDocbaseConnection(DocbaseConnectionFactory.java:26)
at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.createNewConnection(DocbaseConnectionManager.java:192)
at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.getDocbaseConnection(DocbaseConnectionManager.java:122)
at com.documentum.fc.client.impl.session.SessionFactory.newSession(SessionFactory.java:23)
at com.documentum.fc.client.impl.session.PrincipalAwareSessionFactory.newSession(PrincipalAwareSessionFactory.java:44)
at com.documentum.fc.client.impl.session.PooledSessionFactory.newSession(PooledSessionFactory.java:49)
at com.documentum.fc.client.impl.session.SessionManager.getSessionFromFactory(SessionManager.java:134)
at com.documentum.fc.client.impl.session.SessionManager.newSession(SessionManager.java:72)
at com.documentum.fc.client.impl.session.SessionManager.getSession(SessionManager.java:191)
at com.documentum.fc.client.impl.bof.classmgmt.ModuleManager.connect(ModuleManager.java:397)
at com.documentum.fc.client.impl.bof.classmgmt.ModuleManager.init(ModuleManager.java:352)
at com.documentum.fc.client.impl.bof.classmgmt.ModuleManager.getInstance(ModuleManager.java:43)
at com.documentum.fc.client.security.impl.DfcIdentityPublisher.<init>(DfcIdentityPublisher.java:44)
at com.documentum.fc.client.security.internal.RegistrationMgr.register(RegistrationMgr.java:34)
at com.documentum.fc.impl.RuntimeContext.<clinit>(RuntimeContext.java:195)
at com.documentum.fc.client.DfClient.<clinit>(DfClient.java:772)
at com.documentum.com.DfClientX.getLocalClient(DfClientX.java:43)
at com.documentum.server.impl.fulltext.indexagent.install.SessionController.createSessionManager(SessionController.java:43)
at com.documentum.server.impl.fulltext.indexagent.install.SessionController.getSessionManager(SessionController.java:20)
at com.documentum.server.impl.fulltext.indexagent.install.SessionController.getSession(SessionController.java:61)
at com.documentum.server.impl.fulltext.indexagent.install.DocbaseSetup.<init>(DocbaseSetup.java:71)
at com.documentum.install.indexagent.services.Docbase.<init>(Docbase.java:27)
at com.documentum.install.indexagent.services.Docbase.<init>(Docbase.java:22)
at com.documentum.install.indexagent.services.Docbase.createFulltextObjectsUsingMacros(Docbase.java:72)
at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.configureFulltextDocbaseObjects(ConfigIndexAgentActions.java:367)
at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.createIndexAgent(ConfigIndexAgentActions.java:107)
at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.install(ConfigIndexAgentActions.java:50)
at com.zerog.ia.installer.actions.CustomAction.installSelf(Unknown Source)
at com.zerog.ia.installer.InstallablePiece.install(Unknown Source)
at com.zerog.ia.installer.InstallablePiece.install(Unknown Source)
at com.zerog.ia.installer.GhostDirectory.install(Unknown Source)
at com.zerog.ia.installer.InstallablePiece.install(Unknown Source)
at com.zerog.ia.installer.Installer.install(Unknown Source)
at com.zerog.ia.installer.LifeCycleManager.consoleInstallMain(Unknown Source)
at com.zerog.ia.installer.LifeCycleManager.executeApplication(Unknown Source)
at com.zerog.ia.installer.Main.main(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.zerog.lax.LAX.launch(Unknown Source)
at com.zerog.lax.LAX.main(Unknown Source)
09:33:17,373 INFO [main] com.documentum.fc.client.security.impl.JKSKeystoreUtilForDfc - keystore file name is $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/dfc.keystore
09:33:17,395 ERROR [main] com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions - Failed to configure fulltext repository objects.
java.lang.Exception: java.lang.ExceptionInInitializerError
at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.configureFulltextDocbaseObjects(ConfigIndexAgentActions.java:372)
at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.createIndexAgent(ConfigIndexAgentActions.java:107)
at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.install(ConfigIndexAgentActions.java:50)
at com.zerog.ia.installer.actions.CustomAction.installSelf(Unknown Source)
at com.zerog.ia.installer.InstallablePiece.install(Unknown Source)
at com.zerog.ia.installer.InstallablePiece.install(Unknown Source)
at com.zerog.ia.installer.GhostDirectory.install(Unknown Source)
at com.zerog.ia.installer.InstallablePiece.install(Unknown Source)
at com.zerog.ia.installer.Installer.install(Unknown Source)
at com.zerog.ia.installer.LifeCycleManager.consoleInstallMain(Unknown Source)
at com.zerog.ia.installer.LifeCycleManager.executeApplication(Unknown Source)
at com.zerog.ia.installer.Main.main(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.zerog.lax.LAX.launch(Unknown Source)
at com.zerog.lax.LAX.main(Unknown Source)
Caused by: java.lang.ExceptionInInitializerError
at com.documentum.fc.client.DfClient.<clinit>(DfClient.java:772)
at com.documentum.com.DfClientX.getLocalClient(DfClientX.java:43)
at com.documentum.server.impl.fulltext.indexagent.install.SessionController.createSessionManager(SessionController.java:43)
at com.documentum.server.impl.fulltext.indexagent.install.SessionController.getSessionManager(SessionController.java:20)
at com.documentum.server.impl.fulltext.indexagent.install.SessionController.getSession(SessionController.java:61)
at com.documentum.server.impl.fulltext.indexagent.install.DocbaseSetup.<init>(DocbaseSetup.java:71)
at com.documentum.install.indexagent.services.Docbase.<init>(Docbase.java:27)
at com.documentum.install.indexagent.services.Docbase.<init>(Docbase.java:22)
at com.documentum.install.indexagent.services.Docbase.createFulltextObjectsUsingMacros(Docbase.java:72)
at com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions.configureFulltextDocbaseObjects(ConfigIndexAgentActions.java:367)
... 17 more
Caused by: java.lang.NullPointerException
at com.documentum.fc.client.security.impl.IpAndRcHelper.getRegistration(IpAndRcHelper.java:439)
at com.documentum.fc.client.security.impl.DfcIdentityPublisher.getRegistration(DfcIdentityPublisher.java:475)
at com.documentum.fc.client.security.impl.DfcIdentityPublisher.publish(DfcIdentityPublisher.java:134)
at com.documentum.fc.client.security.impl.DfcIdentityPublisher.publish(DfcIdentityPublisher.java:102)
at com.documentum.fc.client.security.internal.RegistrationMgr.register(RegistrationMgr.java:34)
at com.documentum.fc.impl.RuntimeContext.<clinit>(RuntimeContext.java:195)
... 27 more
09:33:17,396 INFO [main] - The INSTALLER_UI value is SILENT
09:33:17,396 INFO [main] - The common.installLocation value is $XPLORE_HOME
09:33:17,396 INFO [main] - The indexagent.configMode.create value is 1
09:33:17,396 INFO [main] - The indexagent.configMode.upgrade value is 0
09:33:17,396 INFO [main] - The indexagent.configMode.delete value is 0
09:33:17,396 INFO [main] - The indexagent.configMode.create.migration value is 0
09:33:17,397 INFO [main] - The indexagent.ess.host value is ds-0.ds.namespace.svc.cluster.local
09:33:17,397 INFO [main] - The indexagent.ess.port value is 9300
09:33:17,397 INFO [main] - The indexagent.name value is Indexagent_REPO2
09:33:17,397 INFO [main] - The indexagent.FQDN value is ds-0.ds.namespace.svc.cluster.local
09:33:17,397 INFO [main] - The indexagent.instance.port value is 9220
09:33:17,397 INFO [main] - The indexagent.instance.password value is ******
09:33:17,397 INFO [main] - The indexagent.docbase.name value is REPO2
09:33:17,397 INFO [main] - The indexagent.docbase.user value is dmadmin
09:33:17,397 INFO [main] - The indexagent.docbase.password value is ******
09:33:17,398 INFO [main] - The indexagent.connectionBroker.host value is cs.namespace.svc.cluster.local
09:33:17,398 INFO [main] - The indexagent.connectionBroker.port value is 1489
09:33:17,398 INFO [main] - The indexagent.globalRegistryRepository.name value is gr_REPO
09:33:17,398 INFO [main] - The indexagent.globalRegistryRepository.user value is dm_bof_registry
09:33:17,398 INFO [main] - The indexagent.globalRegistryRepository.password value is ******
09:33:17,398 INFO [main] - The indexagent.storage.name value is default
09:33:17,398 INFO [main] - The indexagent.local_content_area value is /data/ds-0/data/export/ia/Indexagent_REPO2
09:33:17,398 INFO [main] - The common.installOwner.password value is ******
09:33:17,398 INFO [main] - The indexagent.delete.objects value is 1
09:33:17,398 INFO [main] - The indexagent.STANDBY_FULLTEXT_USER value is dm_fulltext_index_user_02
09:33:17,398 INFO [main] - The indexagent.STANDBY_FULLTEXT_INDEX_NAME value is REPO2_ftindex_02
09:33:17,399 INFO [main] - The indexagent.STANDBY_FULLTEXT_ENGINE_NAME value is DSearch Fulltext Engine Configuration 02
09:33:17,399 INFO [main] - The env PATH value is: /usr/xpg4/bin:$JAVA_HOME/bin:$JAVA_HOME/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/xplore/.local/bin:/home/xplore/bin:/sbin
09:33:17,445 WARN [main] com.documentum.install.indexagent.installanywhere.actions.RegisterIndexAgentWithWatchdog - Stopping watchdog.
09:33:17,445 WARN [main] com.documentum.install.indexagent.installanywhere.actions.RegisterIndexAgentWithWatchdog - Registering ds-0.ds.namespa_9220_IndexAgent with watchdog.
09:33:17,793 INFO [main] com.emc.ess.watchdog.config.registry.impl.EssMultiWatchdogConfiguration - Begin to add SingleWatchdogConfig with hostname: ds-0.ds.namespace.svc.cluster.local
09:33:17,795 INFO [main] com.emc.ess.watchdog.config.EssWatchdogRegistration - Successfully createdcom.emc.ess.watchdog.common.impl.tasks.SendFailureNotificationTask
09:33:17,795 INFO [main] com.emc.ess.watchdog.config.EssWatchdogRegistration - Successfully created IndexAgent task ds-0.ds.namespa_9220_IndexAgent_HeartBeat for application instance ds-0.ds.namespa_9220_IndexAgent
09:33:17,888 INFO [main] com.emc.ess.watchdog.config.EssWatchdogRegistration - Successfully added configuration for {IndexAgent,ds-0.ds.namespa_9220_IndexAgent} under watchdog host ds-0.ds.namespace.svc.cluster.local
09:33:17,888 WARN [main] com.documentum.install.indexagent.installanywhere.actions.RegisterIndexAgentWithWatchdog - Registration succeeded.
[xplore@ds-0 logs]$
As you probably know, xPlore 16.4 is now quite old and it was still a version that came out-of-the-box with the Oracle JDK 8. The issue above is that the DFC Keystore couldn’t be initialized/created for this new IndexAgent and the reason for that is shown in the log file as “NoSuchAlgorithmException: A JSAFE_SecureRandom object of HMACDRBG256 is not available“. HMACDRBG256 is the default Pseudo-Random Number Generator (PRNG) algorithm for FIPS, so that made me think about the cryptoj and other jars that will be needed for RSA/FIPS.
Therefore, the first thing I did was check the current Java version as well as the status of the RSA/FIPS libraries:
[xplore@ds-0 logs]$ cd $JAVA_HOME/jre/lib/ext
[xplore@ds-0 ext]$
[xplore@ds-0 ext]$ java -version
java version "1.8.0_261"
Java(TM) SE Runtime Environment (build 1.8.0_261-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.261-b12, mixed mode)
[xplore@ds-0 ext]$
[xplore@ds-0 ext]$ ls -ltr
total 27792
-rw-r----- 1 xplore xplore 18493706 Jun 18 2020 jfxrt.jar
-rw-r----- 1 xplore xplore 69304 Jun 18 2020 zipfs.jar
-rw-r----- 1 xplore xplore 280139 Jun 18 2020 sunpkcs11.jar
-rw-r----- 1 xplore xplore 280999 Jun 18 2020 sunjce_provider.jar
-rw-r----- 1 xplore xplore 60361 Jun 18 2020 sunec.jar
-rw-r----- 1 xplore xplore 2024157 Jun 18 2020 nashorn.jar
-rw-r----- 1 xplore xplore 1179462 Jun 18 2020 localedata.jar
-rw-r----- 1 xplore xplore 44516 Jun 18 2020 jaccess.jar
-rw-r----- 1 xplore xplore 8286 Jun 18 2020 dnsns.jar
-rw-r----- 1 xplore xplore 3860563 Jun 18 2020 cldrdata.jar
-rw-r----- 1 xplore xplore 1269 Jun 18 2020 meta-index
[xplore@ds-0 ext]$
As you can see, it looked like someone updated/patched the Oracle JDK 8 (I believe the default that comes with xPlore 16.4 being 1.8.0_152 and not 1.8.0_261) but while doing so, only the base JARs were put in place and not the RSA/FIPS related ones. So, it looks like something was missed in the patching steps. These additional JARs aren’t really needed for the runtime, but they are when trying to install a new IndexAgent because of the DFC Keystore creation, so that probably explained why the customer didn’t see any issues in the few months after the JDK upgrade.
I quickly installed an OOTB xPlore 16.4 environment (with the OOTB JDK 8u152) to be able to list all the “custom” JARs present so that I could take them and restore these into the environment with patched JDK. This is the list of JARs (OOTB + “custom”):
[xplore@ds-0 ext]$ ls -ltr
total 29700
-rw-r----- 1 xplore xplore 18513457 Sep 14 2017 jfxrt.jar
-rw-r----- 1 xplore xplore 68965 Sep 14 2017 zipfs.jar
-rw-r----- 1 xplore xplore 248726 Sep 14 2017 sunpkcs11.jar
-rw-r----- 1 xplore xplore 273952 Sep 14 2017 sunjce_provider.jar
-rw-r----- 1 xplore xplore 41672 Sep 14 2017 sunec.jar
-rw-r----- 1 xplore xplore 2023991 Sep 14 2017 nashorn.jar
-rw-r----- 1 xplore xplore 746 Sep 14 2017 meta-index
-rw-r----- 1 xplore xplore 1179101 Sep 14 2017 localedata.jar
-rw-r----- 1 xplore xplore 44516 Sep 14 2017 jaccess.jar
-rw-r----- 1 xplore xplore 8286 Sep 14 2017 dnsns.jar
-rw-r----- 1 xplore xplore 3860502 Sep 14 2017 cldrdata.jar
-rw-r----- 1 xplore xplore 8046 Oct 15 09:41 util.jar
-rw-r----- 1 xplore xplore 421576 Oct 15 09:41 sslj.jar
-rw-r----- 1 xplore xplore 505314 Oct 15 09:41 jcm.jar
-rw-r----- 1 xplore xplore 506533 Oct 15 09:41 jcmFIPS.jar
-rw-r----- 1 xplore xplore 325610 Oct 15 09:41 jcmandroidfips.jar
-rw-r----- 1 xplore xplore 1548393 Oct 15 09:41 cryptojcommon.jar
-rw-r----- 1 xplore xplore 70096 Oct 15 09:41 cryptojce.jar
-rw-r----- 1 xplore xplore 727472 Oct 15 09:41 certj.jar
[xplore@ds-0 ext]$
As you can see above, there are 11 OOTB JARs inside a Java 8u152 and then xPlore during its installation will deploy an additional 8 JARs for RSA/FIPS. Therefore, it looks like the person that upgraded the JDK forgot to restore these 8 JARs into the new JDK folder. For information, this is documented in the xPlore Patch Notes under the “Upgrading Bundled JDK 8” section.
I restored the 8 “custom” JARs related to RSA/FIPS into the upgraded JDK 8u261 and then started again the xPlore IndexAgent installation, which succeeded this time:
[xplore@ds-0 ~]$ cd $JAVA_HOME/jre/lib/ext
[xplore@ds-0 ext]$
[xplore@ds-0 ext]$ ls -ltr
total 29740
-rw-r----- 1 xplore xplore 18493706 Jun 18 2020 jfxrt.jar
-rw-r----- 1 xplore xplore 69304 Jun 18 2020 zipfs.jar
-rw-r----- 1 xplore xplore 280139 Jun 18 2020 sunpkcs11.jar
-rw-r----- 1 xplore xplore 280999 Jun 18 2020 sunjce_provider.jar
-rw-r----- 1 xplore xplore 60361 Jun 18 2020 sunec.jar
-rw-r----- 1 xplore xplore 2024157 Jun 18 2020 nashorn.jar
-rw-r----- 1 xplore xplore 1179462 Jun 18 2020 localedata.jar
-rw-r----- 1 xplore xplore 44516 Jun 18 2020 jaccess.jar
-rw-r----- 1 xplore xplore 8286 Jun 18 2020 dnsns.jar
-rw-r----- 1 xplore xplore 3860563 Jun 18 2020 cldrdata.jar
-rw-r----- 1 xplore xplore 1269 Jun 18 2020 meta-index
-rw-r----- 1 xplore xplore 8046 Oct 15 10:20 util.jar
-rw-r----- 1 xplore xplore 421576 Oct 15 10:20 sslj.jar
-rw-r----- 1 xplore xplore 505314 Oct 15 10:20 jcm.jar
-rw-r----- 1 xplore xplore 506533 Oct 15 10:20 jcmFIPS.jar
-rw-r----- 1 xplore xplore 325610 Oct 15 10:20 jcmandroidfips.jar
-rw-r----- 1 xplore xplore 1548393 Oct 15 10:20 cryptojcommon.jar
-rw-r----- 1 xplore xplore 70096 Oct 15 10:20 cryptojce.jar
-rw-r----- 1 xplore xplore 727472 Oct 15 10:20 certj.jar
[xplore@ds-0 ext]$
[xplore@ds-0 ext]$ cd $XPLORE_HOME/setup/indexagent/logs
[xplore@ds-0 logs]$
[xplore@ds-0 logs]$ cat install.log
10:50:18,085 INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - The product name is: Index Agent
10:50:18,085 INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - The product version is: 16.4.0000.0197
10:50:18,085 INFO [main] -
10:50:18,126 INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - Done InitializeSharedLibrary ...
10:50:27,513 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - setApplicationServer sharedDfcLibDir is:$XPLORE_HOME/dfc
10:50:27,514 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - getFileFromResource for templates/appserver.properties
10:50:27,596 INFO [main] com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions - starting DctmActions
10:50:32,428 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - user name =
10:50:32,430 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - The batch file: $XPLORE_HOME/temp/installer/wildfly/dctm_tmpcmd0.sh exist? false
10:50:32,430 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - The user home is : /home/xplore
10:50:32,435 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - Executing temporary batch file: $XPLORE_HOME/temp/installer/wildfly/dctm_tmpcmd0.sh for running: $JAVA_HOME/bin/java -cp $WILDFLY_HOME/modules/system/layers/base/emc/documentum/security/main/dfc.jar:$WILDFLY_HOME/modules/system/layers/base/emc/documentum/security/main/aspectjrt.jar:$WILDFLY_HOME/modules/system/layers/base/emc/documentum/security/main/DctmUtils.jar com.documentum.install.appserver.utils.DctmCreatePasswordHash xxxx
10:50:32,867 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - Server DctmServer_Indexagent_REPO2 already exists!
10:50:32,868 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - Deploying to Group Indexagent_REPO2... IndexAgent (IndexAgent.war): does not exist!
10:50:32,872 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/dfc.properties
10:50:32,879 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/log4j.properties
10:50:32,886 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/logback.xml
10:50:32,893 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/indexagent.xml
10:50:32,904 INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - resolving $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/web.xml
10:50:32,914 INFO [main] com.documentum.install.shared.actions.DiActionCopyFilesTarget - copying $XPLORE_HOME/setup/indexagent/tools/templates/aclreplication.sh.template to $XPLORE_HOME/setup/indexagent/tools/aclreplication_for_REPO2.sh
10:50:32,931 INFO [main] com.documentum.install.shared.actions.DiActionCopyFilesTarget - copying $XPLORE_HOME/setup/indexagent/tools/templates/ftintegrity.sh.template to $XPLORE_HOME/setup/indexagent/tools/ftintegrity_for_REPO2.sh
10:50:32,949 INFO [main] com.documentum.install.shared.actions.DiActionSetPermissionTarget - performing chmod 755 $XPLORE_HOME/setup/indexagent/tools/aclreplication_for_REPO2.sh
10:50:32,955 INFO [main] com.documentum.install.shared.actions.DiActionSetPermissionTarget - performing chmod 755 $XPLORE_HOME/setup/indexagent/tools/ftintegrity_for_REPO2.sh
10:50:32,962 INFO [main] com.documentum.install.shared.actions.DiActionResolveMacrosTarget - performing macro resolution on $XPLORE_HOME/setup/indexagent/tools/aclreplication_for_REPO2.sh
10:50:32,978 INFO [main] com.documentum.install.shared.actions.DiActionResolveMacrosTarget - performing macro resolution on $XPLORE_HOME/setup/indexagent/tools/ftintegrity_for_REPO2.sh
10:50:32,989 INFO [main] com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions - Finished DctmActions.
10:50:33,385 INFO [main] com.documentum.fc.client.security.impl.JKSKeystoreUtilForDfc - keystore file name is $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/dfc.keystore
10:50:34,002 INFO [main] com.documentum.fc.client.security.internal.CreateIdentityCredential$MultiFormatPKIKeyPair - generated RSA (2,048-bit strength) mutiformat key pair in 328 ms
10:50:34,028 INFO [main] com.documentum.fc.client.security.internal.CreateIdentityCredential - certificate created for DFC <CN=dfc_ZpcFYVkXHfBSo51C2rp7HXSbxbwa,O=EMC,OU=Documentum> valid from Fri Oct 15 10:45:33 UTC 2021 to Mon Oct 13 10:50:33 UTC 2031:
10:50:34,032 INFO [main] com.documentum.fc.client.security.impl.JKSKeystoreUtilForDfc - keystore file name is $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/dfc.keystore
10:50:34,055 INFO [main] com.documentum.fc.client.security.impl.InitializeKeystoreForDfc - [DFC_SECURITY_IDENTITY_INITIALIZED] Initialized new identity in keystore, DFC alias=dfc, identity=dfc_ZpcFYVkXHfBSo51C2rp7HXSbxbwa
10:50:34,056 INFO [main] com.documentum.fc.client.security.impl.AuthenticationMgrForDfc - identity for authentication is dfc_ZpcFYVkXHfBSo51C2rp7HXSbxbwa
10:50:34,057 INFO [main] com.documentum.fc.impl.RuntimeContext - DFC Version is 16.4.0000.0185
10:50:34,078 INFO [Timer-2] com.documentum.fc.client.impl.bof.cache.ClassCacheManager$CacheCleanupTask - [DFC_BOF_RUNNING_CLEANUP] Running class cache cleanup task
10:50:35,346 INFO [main] com.documentum.fc.client.impl.connection.docbase.DocbaseConnection - Object protocol version 2
10:50:35,402 INFO [main] com.documentum.fc.client.security.internal.AuthenticationMgr - new identity bundle <dfc_ZpcFYVkXHfBSo51C2rp7HXSbxbwa 1634295035 ds-0.ds.namespace.svc.cluster.local YnmJpnz48aXW8QuEkgQm/eHbq9hCgJnUGCxdjmPsUJzEnEr5nSwk3KPQ2WUo5ihIuylu2ihLt5o8mTpsEoIBt177lMnAvtU+nbBtB0ItTnD4Bt1M83IQR+t5JAQSYF7Yk7WYCiAADEcKWno6/JiEgkYM0qjU5HwR0w/zmxDSf2330KImqYRPLQtEdCoW8Z3+SSs2/xJdrcki27of9CKH5BXBaXpGpqFzBCYYy76C+SL5+GUR8srU2Q9d4qo5GrlaLJecyWw1Oy3ZZbxDX/nzn7PmBOemkS8rqHZK5msTW9Z8G7Lt6WfXv6w3qumAYJhJCXnXtQEd0TG2liM5XWGqVg==>
10:50:35,689 INFO [main] com.documentum.fc.client.security.impl.JKSKeystoreUtilForDfc - keystore file name is $WILDFLY_HOME/server/DctmServer_Indexagent_REPO2/deployments/IndexAgent.war/WEB-INF/classes/dfc.keystore
10:50:35,734 INFO [main] com.documentum.fc.client.security.impl.DfcIdentityPublisher - found client registration: false
10:50:38,098 INFO [main] com.documentum.fc.client.privilege.impl.PublicKeyCertificate - stored certificate for CN
10:50:38,262 INFO [main] com.documentum.fc.client.security.impl.IpAndRcHelper - filling in gr_REPO a new record with this persistent certificate:
-----BEGIN CERTIFICATE-----
MIIDHzCCAgcCEBVLERfLzAbZ/gi5w7PVnM4wDQYJKoZIhvcNAQELBQAwTjETMBEG
A1UECwwKRG9jdW1lbnR1bTEMMAoGA1UECgwDRU1DMSkwJwYDVQQDDCBkZmNfWlhT
YnhrWHBjRllWMnJwN0hIZkJTbzUxQ2J3YTAeFw0yMTEwMTUxMDQ1MzNaFw0zMTEw
tvbnbJm2IAOE1a1w/dK+NTltmB4QvOoVAvOkaiuTN6qMmi4zCQRjDssaxfdnn9xV
K9lj+v+lXdjlhkCg8PGIssjaAd+zxiVPK9+so52pt2U9rwSk/ASXZqpa3sERO0ua
CJtlPyHuCXHVB6yrwwgnkyWO0PFyfWTi4DUMKJpQCjXGR17+EkfS2u5dGtEyeiLt
CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCiBaGp0Wh0hp3/xptFspRkPKaDoOB3
phrVNQqiJT3CiqJgFabiIEGS3AOkm85372RaS4ZrqUh++P5LEkflFZuRzvgmeLMj
HAH5VPSKM8QE3VrVQo17X1msfq4HeRXpUYPk7iL9HPoDSrL7J+kphTPrAgMBAAEw
DQYJKoZIhvcNAQELBQADggEBAB4pqKL0J7C63/IyJGXN/jZXaN8Fg2bz5bYythgI
MTMxMDUwMzNaME4xEzARBgNVBAsMCkRvY3VtZW50dW0xDDAKBgNVBAoMA0VNQzEp
MCcGA1UEAwwgZGZjX1pYU2J4a1hwY0ZZVjJycDdISGZCU281MUNid2EwggEiMA0G
ZwF0p0Io6oDVXjK921c7TovGz8SdxVARSR/LG7EAXYAqkvXkXMkow65BonvBPZcE
Y6oYUq7OrtWTWd+yOryZ+y+TwG0P2cDd5nErCSoiuNlJ+MvhvJ/fraknJsr9/iGD
rZvZpAT0aRAKwKaj65RlK6Pj43B6rPlouD1yS6Kgb5KpgOCuK/HxpcseUsmNDh5v
JRD05r0VxD6b3nMqRP3E7XPr3hbEbq79R6qF8aaF+mm4SZNSEgm1Ps2ruzchwShe
Xn+DBv7XA0xCOmHbLvLe/tLlNKH8GPCuqcbcSUTTCmyE8/A=
-----END CERTIFICATE-----
10:50:38,266 INFO [main] com.documentum.fc.client.security.impl.DfcIdentityPublisher - found client registration: false
10:50:38,454 INFO [main] com.documentum.fc.client.security.impl.IpAndRcHelper - filling a new registration record for dfc_ZpcFYVkXHfBSo51C2rp7HXSbxbwa
10:50:38,478 INFO [main] com.documentum.fc.client.security.impl.DfcIdentityPublisher - [DFC_SECURITY_GR_REGISTRATION_PUBLISH] this dfc instance is now published in the global registry gr_REPO
10:50:39,542 INFO [main] com.documentum.fc.client.impl.connection.docbase.DocbaseConnection - Object protocol version 2
10:50:45,788 INFO [main] com.documentum.fc.client.impl.connection.docbase.DocbaseConnection - Object protocol version 2
10:50:46,549 INFO [main] com.documentum.fc.client.impl.connection.docbase.DocbaseConnection - Object protocol version 2
10:50:47,848 INFO [main] com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions - storage name is set to: default
10:50:49,292 INFO [main] com.documentum.install.indexagent.installanywhere.actions.ConfigIndexAgentActions - creating $XPLORE_HOME/installinfo/instances/indexagent
10:50:49,332 WARN [main] com.documentum.install.indexagent.installanywhere.actions.RegisterIndexAgentWithWatchdog - Stopping watchdog.
10:50:49,332 WARN [main] com.documentum.install.indexagent.installanywhere.actions.RegisterIndexAgentWithWatchdog - Registering ds-0.ds.namespa_9220_IndexAgent with watchdog.
10:50:49,568 INFO [main] com.emc.ess.watchdog.config.registry.impl.EssMultiWatchdogConfiguration - Begin to add SingleWatchdogConfig with hostname: ds-0.ds.namespace.svc.cluster.local
10:50:49,568 INFO [main] com.emc.ess.watchdog.config.EssWatchdogRegistration - Successfully added configuration for {IndexAgent,ds-0.ds.namespa_9220_IndexAgent} under watchdog host ds-0.ds.namespace.svc.cluster.local
10:50:49,568 WARN [main] com.documentum.install.indexagent.installanywhere.actions.RegisterIndexAgentWithWatchdog - Registration succeeded.
[xplore@ds-0 logs]$
Long story short, whenever you want to try to patch something, always check the documentation to make sure you aren’t forgetting something! Even if the system can start and run properly after the patching, you might cause some issues to other people in the future like it was the case here ;).
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/MOP_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/PLE_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/ATR_web-min-scaled.jpg)