{"id":17939,"date":"2022-07-15T21:05:00","date_gmt":"2022-07-15T19:05:00","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/?p=17939"},"modified":"2022-07-15T16:18:06","modified_gmt":"2022-07-15T14:18:06","slug":"documentum-xplore-not-able-to-deploy-dsearch-war","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/","title":{"rendered":"Documentum – xPlore not able to deploy “dsearch.war”"},"content":{"rendered":"\n

Recently at a customer, I faced a new issue I didn’t see before on the xPlore Dsearch component. Basically, the processes appeared to be starting and the “dsearchadmin.war” was deployed properly but it always failed for the “dsearch.war” upon startup of WildFly. However, it was possible to “deploy” it once WildFly was running\u2026 This issue wasn’t too complex to debug because some interesting hints were displayed on the logs but it’s an interesting behavior because you could think that everything is running while it’s not. The initial issue was the following one:<\/p>\n\n\n

\n[xplore@ds-0 ~]$ cd $JBOSS_HOME\/server\n[xplore@ds-0 server]$\n[xplore@ds-0 server]$ $STARTSTOP start\n  **\n  **  The PrimaryDsearch is shutdown\n  **\nINFO - Starting the PrimaryDsearch...\n  **\n  **  The PrimaryDsearch is running with PID: 9178\n  **\n[xplore@ds-0 server]$\n[xplore@ds-0 server]$ ps uxf\nUSER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND\nxplore    9075  0.0  0.0  13428  1552 pts\/1    S    12:06   0:00 \/bin\/sh .\/startPrimaryDsearch.sh\nxplore    9077  0.0  0.0  13448  1744 pts\/1    S    12:06   0:00  \\_ \/bin\/sh $XPLORE_HOME\/wildfly17.0.1\/bin\/standalone.sh\nxplore    9178 83.2  1.1 10193500 1580936 pts\/1 Sl  12:06   1:32      \\_ $XPLORE_HOME\/java64\/JAVA_LINK\/bin\/java -D[Standalone] -server -Xms8g -Xmx8g -XX:MaxMetaspaceSize=512m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:G1HeapRegionSize=4m -Djbo\nxplore    1401  0.0  0.0  13568  2260 pts\/2    Ss   11:50   0:00 bash -l\nxplore    9940  0.0  0.0  53336  1864 pts\/2    R+   12:07   0:00  \\_ ps uxf\n[xplore@ds-0 server]$\n[xplore@ds-0 server]$ cat DctmServer_PrimaryDsearch\/log\/server.log\n...\n...\n2022-07-13 12:06:30,311 UTC INFO  [org.jboss.ws.cxf.deployment] (MSC service thread 1-2) JBWS024074: WSDL published to: file:$XPLORE_HOME\/wildfly17.0.1\/server\/DctmServer_PrimaryDsearch\/data\/wsdl\/dsearch.war\/ESSAdminWebServiceService.wsdl\n2022-07-13 12:06:30,356 UTC WARN  [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0274: Excluded dependency org.slf4j.impl via jboss-deployment-structure.xml does not exist.\n2022-07-13 12:06:30,356 UTC INFO  [org.jboss.as.webservices] (MSC service thread 1-1) WFLYWS0003: Starting service jboss.ws.endpoint."dsearch.war"."com.emc.documentum.core.fulltext.indexserver.admin.controller.ESSAdminWebService"\n2022-07-13 12:06:30,421 UTC WARN  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0101: Duplicate servlet mapping \/ESSAdminWebService found\n2022-07-13 12:06:34,847 UTC INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 67) WFLYUT0021: Registered web context: '\/dsearchadmin' for server 'default-server'\n2022-07-13 12:06:34,996 UTC INFO  [javax.enterprise.resource.webcontainer.jsf.config] (ServerService Thread Pool -- 74) Initializing Mojarra 2.3.9.SP02 for context '\/dsearch'\n2022-07-13 12:06:40,012 UTC ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 74) MSC000001: Failed to start service jboss.deployment.unit."dsearch.war".undertow-deployment: org.jboss.msc.service.StartException in service jboss.deployment.unit."dsearch.war".undertow-deployment: java.lang.ExceptionInInitializerError\n        at org.wildfly.extension.undertow@17.0.1.Final\/\/org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81)\n        at java.base\/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n        at java.base\/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n        at org.jboss.threads@2.3.3.Final\/\/org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)\n        at org.jboss.threads@2.3.3.Final\/\/org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)\n        at org.jboss.threads@2.3.3.Final\/\/org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)\n        at org.jboss.threads@2.3.3.Final\/\/org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)\n        at java.base\/java.lang.Thread.run(Thread.java:829)\n        at org.jboss.threads@2.3.3.Final\/\/org.jboss.threads.JBossThread.run(JBossThread.java:485)\nCaused by: java.lang.ExceptionInInitializerError\n        at deployment.dsearch.war\/\/com.emc.cma.cps.management.CPSManager.shutdownManager(CPSManager.java:806)\n        at deployment.dsearch.war\/\/com.emc.cma.cps.management.CPSManager.terminate(CPSManager.java:160)\n        at deployment.dsearch.war\/\/com.emc.cma.cps.management.CPSManager.init(CPSManager.java:1019)\n        at deployment.dsearch.war\/\/com.emc.cma.cps.management.CPSManager.getInstance(CPSManager.java:114)\n        at deployment.dsearch.war\/\/com.emc.cma.cps.services.CPSServiceImpl.getVersion(CPSServiceImpl.java:46)\n        at deployment.dsearch.war\/\/com.emc.cma.cps.rt.ContentProcessingServiceLocalStub.getVersion(ContentProcessingServiceLocalStub.java:56)\n        at deployment.dsearch.war\/\/com.emc.documentum.core.fulltext.indexserver.cps.CPSSubmitter.connectCPS(CPSSubmitter.java:301)\n        at deployment.dsearch.war\/\/com.emc.documentum.core.fulltext.indexserver.cps.CPSSubmitter.<init>(CPSSubmitter.java:159)\n        at deployment.dsearch.war\/\/com.emc.documentum.core.fulltext.indexserver.cps.CPSRouter.addCPS(CPSRouter.java:108)\n        at deployment.dsearch.war\/\/com.emc.documentum.core.fulltext.indexserver.cps.CPSRouter.<init>(CPSRouter.java:85)\n        at deployment.dsearch.war\/\/com.emc.documentum.core.fulltext.indexserver.core.ESSContext.initialize(ESSContext.java:240)\n        at deployment.dsearch.war\/\/com.emc.documentum.core.fulltext.indexserver.core.ESSNode.startUp(ESSNode.java:67)\n        at deployment.dsearch.war\/\/com.emc.documentum.core.fulltext.webapp.IndexServerServlet.init(IndexServerServlet.java:48)\n        at io.undertow.servlet@2.0.21.Final\/\/io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)\n        at org.wildfly.extension.undertow@17.0.1.Final\/\/org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78)\n        at io.undertow.servlet@2.0.21.Final\/\/io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)\n        at io.undertow.servlet@2.0.21.Final\/\/io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:303)\n        at io.undertow.servlet@2.0.21.Final\/\/io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:143)\n        at io.undertow.servlet@2.0.21.Final\/\/io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:583)\n        at io.undertow.servlet@2.0.21.Final\/\/io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:554)\n        at io.undertow.servlet@2.0.21.Final\/\/io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)\n        at io.undertow.servlet@2.0.21.Final\/\/io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)\n        at org.wildfly.extension.undertow@17.0.1.Final\/\/org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)\n        at org.wildfly.extension.undertow@17.0.1.Final\/\/org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)\n        at org.wildfly.extension.undertow@17.0.1.Final\/\/org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)\n        at org.wildfly.extension.undertow@17.0.1.Final\/\/org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)\n        at org.wildfly.extension.undertow@17.0.1.Final\/\/org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)\n        at io.undertow.servlet@2.0.21.Final\/\/io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:596)\n        at org.wildfly.extension.undertow@17.0.1.Final\/\/org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:97)\n        at org.wildfly.extension.undertow@17.0.1.Final\/\/org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:78)\n        ... 8 more\nCaused by: java.lang.NullPointerException\n        at java.base\/sun.nio.fs.UnixPath.normalizeAndCheck(UnixPath.java:75)\n        at java.base\/sun.nio.fs.UnixPath.<init>(UnixPath.java:69)\n        at java.base\/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:279)\n        at java.base\/java.nio.file.Path.of(Path.java:147)\n        at java.base\/java.nio.file.Paths.get(Paths.java:69)\n        at deployment.dsearch.war\/\/com.emc.cma.cps.util.FileUtils.isAbsoluteDirectory(FileUtils.java:8)\n        at deployment.dsearch.war\/\/com.emc.cma.cps.management.CPSConfiguration.getEffectiveTempDirectory(CPSConfiguration.java:394)\n        at deployment.dsearch.war\/\/com.emc.cma.cps.processor.common.CPSContentBufferManager.<clinit>(CPSContentBufferManager.java:166)\n        ... 38 more\n\n2022-07-13 12:06:40,020 UTC ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "dsearch.war")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.unit.\\"dsearch.war\\".undertow-deployment" => "java.lang.ExceptionInInitializerError\n    Caused by: java.lang.ExceptionInInitializerError\n    Caused by: java.lang.NullPointerException"}}\n2022-07-13 12:06:41,370 UTC INFO  [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0010: Deployed "dsearchadmin.war" (runtime-name : "dsearchadmin.war")\n2022-07-13 12:06:41,370 UTC INFO  [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0010: Deployed "dsearch.war" (runtime-name : "dsearch.war")\n2022-07-13 12:06:41,373 UTC INFO  [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report\nWFLYCTL0186:   Services which failed to start:      service jboss.deployment.unit."dsearch.war".undertow-deployment: java.lang.ExceptionInInitializerError\n\n2022-07-13 12:06:41,462 UTC INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server\n2022-07-13 12:06:41,468 UTC ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: WildFly Full 17.0.1.Final (WildFly Core 9.0.2.Final) started (with errors) in 36174ms - Started 1074 of 1323 services (2 services failed or missing dependencies, 400 services are lazy, passive or on-demand)\n[xplore@ds-0 server]$\n<\/pre><\/div>\n\n\n
As you can see above, there are no cps_daemon child processes for the PrimaryDsearch, which is a good indication that even if the WildFly process is running, the xPlore PrimaryDsearch isn’t fully up&running and that something went wrong. On the logs, and especially on the stack trace above, the issue appears to be on the initialization\/startup of the PrimaryDsearch and more specifically around its included CPS. Looking at the URL obviously fails since the application has been switched to failed state so the first thing I tried was simply to test a redeploy of the application. The custom script $STARTSTOP<\/em> we are using (c.f. above) already take care to force a redeployment, in case one of the Applications is in “undeployed<\/em>” or “failed<\/em>” state before WildFly starts, but I still tried to do it manually during runtime, to see if it would work:<\/p>\n\n\n
\n[xplore@ds-0 server]$ curl -kI http:\/\/`hostname -f`:9300\/dsearch\nHTTP\/1.1 404 Not Found\nConnection: keep-alive\nServer: FT1\nContent-Length: 74\nContent-Type: text\/html\nDate: Wed, 13 Jul 2022 12:07:36 GMT\n[xplore@ds-0 server]$\n[xplore@ds-0 server]$ cd DctmServer_PrimaryDsearch\/deployments\/\n[xplore@ds-0 deployments]$\n[xplore@ds-0 deployments]$ ls -l\ntotal 24\ndrwxrwx--- 11 xplore xplore 4096 Jul 13 07:17 dsearchadmin.war\n-rw-r-----  1 xplore xplore   16 Jul 13 07:17 dsearchadmin.war.deployed\ndrwxrwx---  4 xplore xplore 4096 Jul 13 07:17 dsearch.war\n-rw-r-----  1 xplore xplore  251 Jul 13 12:06 dsearch.war.failed\n-rw-rw----  1 xplore xplore 9078 Jul 13 07:17 README.txt\n[xplore@ds-0 deployments]$\n[xplore@ds-0 deployments]$ mv dsearch.war.failed dsearch.war.dodeploy\n[xplore@ds-0 deployments]$\n[xplore@ds-0 deployments]$ ls -l\ntotal 24\ndrwxrwx--- 11 xplore xplore 4096 Jul 13 07:17 dsearchadmin.war\n-rw-r-----  1 xplore xplore   16 Jul 13 07:17 dsearchadmin.war.deployed\ndrwxrwx---  4 xplore xplore 4096 Jul 13 07:17 dsearch.war\n-rw-r-----  1 xplore xplore  251 Jul 13 12:06 dsearch.war.dodeploy\n-rw-r-----  1 xplore xplore   11 Jul 13 12:08 dsearch.war.isdeploying\n-rw-rw----  1 xplore xplore 9078 Jul 13 07:17 README.txt\n[xplore@ds-0 deployments]$\n[xplore@ds-0 deployments]$ sleep 20\n[xplore@ds-0 deployments]$\n[xplore@ds-0 deployments]$ ls -l\ntotal 20\ndrwxrwx--- 11 xplore xplore 4096 Jul 13 07:17 dsearchadmin.war\n-rw-r-----  1 xplore xplore   16 Jul 13 07:17 dsearchadmin.war.deployed\ndrwxrwx---  4 xplore xplore 4096 Jul 13 07:17 dsearch.war\n-rw-r-----  1 xplore xplore   11 Jul 13 07:18 dsearch.war.deployed\n-rw-rw----  1 xplore xplore 9078 Jul 13 07:17 README.txt\n[xplore@ds-0 deployments]$\n<\/pre><\/div>\n\n\n
Somehow, it looked like WildFly was able to deploy the dsearch.war application this time\u2026 So, all happy, I looked at the server.log again to confirm that:<\/p>\n\n\n
\n[xplore@ds-0 deployments]$ cat ..\/log\/server.log\n...\n2022-07-13 12:06:40,020 UTC ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "dsearch.war")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.unit.\\"dsearch.war\\".undertow-deployment" => "java.lang.ExceptionInInitializerError\n    Caused by: java.lang.ExceptionInInitializerError\n    Caused by: java.lang.NullPointerException"}}\n2022-07-13 12:06:41,370 UTC INFO  [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0010: Deployed "dsearchadmin.war" (runtime-name : "dsearchadmin.war")\n2022-07-13 12:06:41,370 UTC INFO  [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0010: Deployed "dsearch.war" (runtime-name : "dsearch.war")\n2022-07-13 12:06:41,373 UTC INFO  [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report\nWFLYCTL0186:   Services which failed to start:      service jboss.deployment.unit."dsearch.war".undertow-deployment: java.lang.ExceptionInInitializerError\n\n2022-07-13 12:06:41,462 UTC INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server\n2022-07-13 12:06:41,468 UTC ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: WildFly Full 17.0.1.Final (WildFly Core 9.0.2.Final) started (with errors) in 36174ms - Started 1074 of 1323 services (2 services failed or missing dependencies, 400 services are lazy, passive or on-demand)\n2022-07-13 12:08:51,775 UTC INFO  [org.jboss.as.webservices] (MSC service thread 1-5) WFLYWS0004: Stopping service jboss.ws.endpoint."dsearch.war"."com.emc.documentum.core.fulltext.indexserver.admin.controller.ESSAdminWebService"\n2022-07-13 12:08:52,508 UTC INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0028: Stopped deployment dsearch.war (runtime-name: dsearch.war) in 739ms\n2022-07-13 12:08:52,517 UTC INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0027: Starting deployment of "dsearch.war" (runtime-name: "dsearch.war")\n...\n...\n2022-07-13 12:09:03,839 UTC INFO  [org.jboss.ws.cxf.deployment] (MSC service thread 1-1) JBWS024074: WSDL published to: file:$XPLORE_HOME\/wildfly17.0.1\/server\/DctmServer_PrimaryDsearch\/data\/wsdl\/dsearch.war\/ESSAdminWebServiceService.wsdl\n2022-07-13 12:09:03,884 UTC INFO  [org.jboss.as.webservices] (MSC service thread 1-1) WFLYWS0003: Starting service jboss.ws.endpoint."dsearch.war"."com.emc.documentum.core.fulltext.indexserver.admin.controller.ESSAdminWebService"\n2022-07-13 12:09:03,884 UTC WARN  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0274: Excluded dependency org.slf4j.impl via jboss-deployment-structure.xml does not exist.\n2022-07-13 12:09:03,908 UTC WARN  [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0101: Duplicate servlet mapping \/ESSAdminWebService found\n2022-07-13 12:09:07,181 UTC INFO  [javax.enterprise.resource.webcontainer.jsf.config] (ServerService Thread Pool -- 80) Initializing Mojarra 2.3.9.SP02 for context '\/dsearch'\n2022-07-13 12:09:10,106 UTC INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 80) WFLYUT0021: Registered web context: '\/dsearch' for server 'default-server'\n2022-07-13 12:09:10,168 UTC INFO  [org.jboss.as.server] (DeploymentScanner-threads - 2) WFLYSRV0016: Replaced deployment "dsearch.war" with deployment "dsearch.war"\n[xplore@ds-0 deployments]$\n<\/pre><\/div>\n\n\n
As you can see above, the initial deployment failed (12:06:40) but then, by manually asking for a redeployment (12:08:51), it looks like WildFly was able to successfully deploy it, no? Let’s look at the URL:<\/p>\n\n\n
\n[xplore@ds-0 deployments]$ curl -kI http:\/\/`hostname -f`:9300\/dsearch\nHTTP\/1.1 302 Found\nConnection: keep-alive\nServer: FT1\nLocation: http:\/\/ds-0.domain.com:9300\/dsearch\/\nContent-Length: 0\nDate: Wed, 13 Jul 2022 12:10:01 GMT\n\n[xplore@ds-0 deployments]$\n<\/pre><\/div>\n\n\n
The URL is now responding with a correct “302 Found” which means that it looks good, right? To cross-check, I just added the “\/” at the end to use the real redirected location:<\/p>\n\n\n
\n[xplore@ds-0 deployments]$ curl -kI http:\/\/`hostname -f`:9300\/dsearch\/\nHTTP\/1.1 404 Not Found\nConnection: keep-alive\nServer: FT1\nContent-Length: 0\nDate: Wed, 13 Jul 2022 12:10:05 GMT\n\n[xplore@ds-0 deployments]$\n<\/pre><\/div>\n\n\n
That’s very peculiar, the deployment appears to be successful, and the first URL appears to work as well but if you really look behind the first link, it’s not working. Of course, the cps_daemon child processes of the PrimaryDsearch are also not present. Therefore, you might think that it started but it didn’t. As mentioned previously, the stack trace appeared to be linked to CPS initialization. Therefore, I looked at the CPS configuration file which is, by default, $XPLORE_HOME\/dsearch\/cps\/cps_daemon\/###DS_NAME###_local_configuration.xml<\/em> for a Dsearch or $XPLORE_HOME\/dsearch\/cps\/cps_daemon\/###CPS_NAME###_configuration.xml<\/em> for a CPS Only. In any cases, you can find the correct name\/path of this file in the “cps.properties<\/em>” file (for a CPS Only, it’s inside the cps.war, not inside the dsearch.war):<\/p>\n\n\n
\n[xplore@ds-0 deployments]$ cps_conf_file=$(grep cps.configuration.file dsearch.war\/WEB-INF\/classes\/cps.properties | awk -F= '{print $2}')\n[xplore@ds-0 deployments]$\n[xplore@ds-0 deployments]$ echo ${cps_conf_file}\n$XPLORE_HOME\/dsearch\/cps\/cps_daemon\/PrimaryDsearch_local_configuration.xml\n[xplore@ds-0 deployments]$\n[xplore@ds-0 deployments]$ ls -l ${cps_conf_file}\nls: cannot access $XPLORE_HOME\/dsearch\/cps\/cps_daemon\/PrimaryDsearch_local_configuration.xml: No such file or directory\n[xplore@ds-0 deployments]$\n<\/pre><\/div>\n\n\n
It appears that the CPS configuration file is gone and that is what is causing this issue, since these processes cannot be initialized. This file was unfortunately removed by mistake a few hours earlier and because of that, the xPlore processes couldn’t start anymore. Restoring the file (from the closest backup) and then restarting xPlore processes was sufficient to have the services back online (since the “\/dsearch” isn’t really reachable, there are obviously errors during the shutdown):<\/p>\n\n\n
\n[xplore@ds-0 deployments]$ $STARTSTOP stop\n  **\n  **  The PrimaryDsearch is running with PID: 9178\n  **\nINFO - Stopping the PrimaryDsearch...\nInstance {PrimaryDsearch} is about to shut down, wait for shutdown complete message.\nException in thread "main" java.lang.IllegalArgumentException: Fail to connect remote server ...\n{\n    "outcome" => "success",\n    "result" => undefined\n}\n  **\n  **  The PrimaryDsearch is shutdown\n  **\n[xplore@ds-0 deployments]$\n[xplore@ds-0 deployments]$ # Restoring the file from the closest backup. Post restore:\n[xplore@ds-0 deployments]$ ls -l ${cps_conf_file}\n-rw-rw---- 1 xplore xplore 9269 Jul 13 12:22 $XPLORE_HOME\/dsearch\/cps\/cps_daemon\/PrimaryDsearch_local_configuration.xml\n[xplore@ds-0 deployments]$\n[xplore@ds-0 deployments]$ $STARTSTOP start\n  **\n  **  The PrimaryDsearch is shutdown\n  **\nINFO - Starting the PrimaryDsearch...\n  **\n  **  The PrimaryDsearch is running with PID: 14826\n  **\n[xplore@ds-0 deployments]$\n<\/pre><\/div>\n\n\n
And cross-checking with the logs and URLs, it started properly:<\/p>\n\n\n
\n[xplore@ds-0 deployments]$ cat ..\/log\/server.log\n...\n2022-07-13 12:23:08,670 UTC INFO  [org.jboss.ws.cxf.deployment] (MSC service thread 1-2) JBWS024074: WSDL published to: file:$XPLORE_HOME\/wildfly17.0.1\/server\/DctmServer_PrimaryDsearch\/data\/wsdl\/dsearch.war\/ESSAdminWebServiceService.wsdl\n2022-07-13 12:23:08,714 UTC INFO  [org.jboss.as.webservices] (MSC service thread 1-6) WFLYWS0003: Starting service jboss.ws.endpoint."dsearch.war"."com.emc.documentum.core.fulltext.indexserver.admin.controller.ESSAdminWebService"\n2022-07-13 12:23:08,715 UTC WARN  [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0274: Excluded dependency org.slf4j.impl via jboss-deployment-structure.xml does not exist.\n2022-07-13 12:23:08,755 UTC WARN  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0101: Duplicate servlet mapping \/ESSAdminWebService found\n2022-07-13 12:23:11,846 UTC INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 76) WFLYUT0021: Registered web context: '\/dsearchadmin' for server 'default-server'\n2022-07-13 12:23:13,634 UTC INFO  [javax.enterprise.resource.webcontainer.jsf.config] (ServerService Thread Pool -- 93) Initializing Mojarra 2.3.9.SP02 for context '\/dsearch'\n2022-07-13 12:23:27,323 UTC INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 93) WFLYUT0021: Registered web context: '\/dsearch' for server 'default-server'\n2022-07-13 12:23:27,356 UTC INFO  [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0010: Deployed "dsearch.war" (runtime-name : "dsearch.war")\n2022-07-13 12:23:27,357 UTC INFO  [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0010: Deployed "dsearchadmin.war" (runtime-name : "dsearchadmin.war")\n2022-07-13 12:23:27,607 UTC INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server\n2022-07-13 12:23:27,614 UTC INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 17.0.1.Final (WildFly Core 9.0.2.Final) started in 44869ms - Started 1077 of 1323 services (400 services are lazy, passive or on-demand)\n[xplore@ds-0 deployments]$\n[xplore@ds-0 deployments]$ curl -kI http:\/\/`hostname -f`:9300\/dsearch\nHTTP\/1.1 302 Found\nConnection: keep-alive\nServer: FT1\nLocation: http:\/\/ds-0.domain.com:9300\/dsearch\/\nContent-Length: 0\nDate: Wed, 13 Jul 2022 12:26:05 GMT\n\n[xplore@ds-0 deployments]$\n[xplore@ds-0 deployments]$ curl -kI http:\/\/`hostname -f`:9300\/dsearch\/\nHTTP\/1.1 259 Unknown\nConnection: keep-alive\nServer: FT1\nContent-Type: text\/html;charset=UTF-8\nContent-Length: 66\nDate: Wed, 13 Jul 2022 12:26:07 GMT\n\n[xplore@ds-0 deployments]$\n<\/pre><\/div>\n\n\n
The HTTP response code of 259 is normal for the dsearch URL so it shows that it works properly now. All cps_daemon child processes are also present and there are no issues on any logs. So just to conclude on this blog, it’s not because it appears that WildFly properly deployed the xPlore applications that they are really working! Make sure to always check the URLs and processes, that’s always a better indication. The best is of course to test xPlore end-to-end from Repository Searches & Indexing but that can take some time if it’s not automated\/scripted.<\/p>\n","protected":false},"excerpt":{"rendered":"
Recently at a customer, I faced a new issue I didn’t see before on the xPlore Dsearch component. Basically, the processes appeared to be starting and the “dsearchadmin.war” was deployed properly but it always failed for the “dsearch.war” upon startup of WildFly. However, it was possible to “deploy” it once WildFly was running\u2026 This issue […]<\/p>\n","protected":false},"author":20,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[197,525],"tags":[2609,912,2610,1100,474],"type_dbi":[],"class_list":["post-17939","post","type-post","status-publish","format-standard","hentry","category-application-integration-middleware","category-enterprise-content-management","tag-documentum-2","tag-dsearch","tag-jboss-2","tag-wildfly","tag-xplore"],"acf":[],"yoast_head":"\nDocumentum - xPlore not able to deploy "dsearch.war" - dbi Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Documentum - xPlore not able to deploy "dsearch.war"\" \/>\n<meta property=\"og:description\" content=\"Recently at a customer, I faced a new issue I didn’t see before on the xPlore Dsearch component. Basically, the processes appeared to be starting and the “dsearchadmin.war” was deployed properly but it always failed for the “dsearch.war” upon startup of WildFly. However, it was possible to “deploy” it once WildFly was running\u2026 This issue […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2022-07-15T19:05:00+00:00\" \/>\n<meta name=\"author\" content=\"Morgan Patou\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@MorganPatou\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Morgan Patou\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/\"},\"author\":{\"name\":\"Morgan Patou\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/c4d05b25843a9bc2ab20415dae6bd2d8\"},\"headline\":\"Documentum – xPlore not able to deploy “dsearch.war”\",\"datePublished\":\"2022-07-15T19:05:00+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/\"},\"wordCount\":683,\"commentCount\":0,\"keywords\":[\"Documentum\",\"Dsearch\",\"JBoss\",\"WildFly\",\"xPlore\"],\"articleSection\":[\"Application integration & Middleware\",\"Enterprise content management\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/\",\"name\":\"Documentum - xPlore not able to deploy \\\"dsearch.war\\\" - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"datePublished\":\"2022-07-15T19:05:00+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/c4d05b25843a9bc2ab20415dae6bd2d8\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Documentum – xPlore not able to deploy “dsearch.war”\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/c4d05b25843a9bc2ab20415dae6bd2d8\",\"name\":\"Morgan Patou\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/5d7f5bec8b597db68a09107a6f5309e3870d6296ef94fb10ead4b09454ca67e5?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/5d7f5bec8b597db68a09107a6f5309e3870d6296ef94fb10ead4b09454ca67e5?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/5d7f5bec8b597db68a09107a6f5309e3870d6296ef94fb10ead4b09454ca67e5?s=96&d=mm&r=g\",\"caption\":\"Morgan Patou\"},\"description\":\"Morgan Patou has over 12 years of experience in Enterprise Content Management (ECM) systems, with a strong focus in recent years on platforms such as Alfresco, Documentum, and M-Files. He specializes in the architecture, setup, customization, and maintenance of ECM infrastructures in complex & critical environments. Morgan is well-versed in both engineering and operations aspects, including high availability design, system integration, and lifecycle management. He also has a solid foundation in open-source and proprietary technologies - ranging from Apache, OpenLDAP or Kerberos to enterprise-grade systems like WebLogic. Morgan Patou holds an Engineering Degree in Computer Science from ENSISA (\u00c9cole Nationale Sup\u00e9rieure d'Ing\u00e9nieurs Sud Alsace) in Mulhouse, France. He is Alfresco Content Services Certified Administrator (ACSCA), Alfresco Content Services Certified Engineer (ACSCE) as well as OpenText Documentum Certified Administrator. His industry experience spans the Public Sector, IT Services, Financial Services\/Banking, and the Pharmaceutical industry.\",\"sameAs\":[\"https:\/\/blog.dbi-services.com\/author\/morgan-patou\/\",\"https:\/\/x.com\/MorganPatou\"],\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/morgan-patou\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Documentum - xPlore not able to deploy \"dsearch.war\" - dbi Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/","og_locale":"en_US","og_type":"article","og_title":"Documentum - xPlore not able to deploy \"dsearch.war\"","og_description":"Recently at a customer, I faced a new issue I didn’t see before on the xPlore Dsearch component. Basically, the processes appeared to be starting and the “dsearchadmin.war” was deployed properly but it always failed for the “dsearch.war” upon startup of WildFly. However, it was possible to “deploy” it once WildFly was running\u2026 This issue […]","og_url":"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/","og_site_name":"dbi Blog","article_published_time":"2022-07-15T19:05:00+00:00","author":"Morgan Patou","twitter_card":"summary_large_image","twitter_creator":"@MorganPatou","twitter_misc":{"Written by":"Morgan Patou","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/"},"author":{"name":"Morgan Patou","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/c4d05b25843a9bc2ab20415dae6bd2d8"},"headline":"Documentum – xPlore not able to deploy “dsearch.war”","datePublished":"2022-07-15T19:05:00+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/"},"wordCount":683,"commentCount":0,"keywords":["Documentum","Dsearch","JBoss","WildFly","xPlore"],"articleSection":["Application integration & Middleware","Enterprise content management"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/","url":"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/","name":"Documentum - xPlore not able to deploy \"dsearch.war\" - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2022-07-15T19:05:00+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/c4d05b25843a9bc2ab20415dae6bd2d8"},"breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/documentum-xplore-not-able-to-deploy-dsearch-war\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Documentum – xPlore not able to deploy “dsearch.war”"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/c4d05b25843a9bc2ab20415dae6bd2d8","name":"Morgan Patou","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/5d7f5bec8b597db68a09107a6f5309e3870d6296ef94fb10ead4b09454ca67e5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/5d7f5bec8b597db68a09107a6f5309e3870d6296ef94fb10ead4b09454ca67e5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/5d7f5bec8b597db68a09107a6f5309e3870d6296ef94fb10ead4b09454ca67e5?s=96&d=mm&r=g","caption":"Morgan Patou"},"description":"Morgan Patou has over 12 years of experience in Enterprise Content Management (ECM) systems, with a strong focus in recent years on platforms such as Alfresco, Documentum, and M-Files. He specializes in the architecture, setup, customization, and maintenance of ECM infrastructures in complex & critical environments. Morgan is well-versed in both engineering and operations aspects, including high availability design, system integration, and lifecycle management. He also has a solid foundation in open-source and proprietary technologies - ranging from Apache, OpenLDAP or Kerberos to enterprise-grade systems like WebLogic. Morgan Patou holds an Engineering Degree in Computer Science from ENSISA (\u00c9cole Nationale Sup\u00e9rieure d'Ing\u00e9nieurs Sud Alsace) in Mulhouse, France. He is Alfresco Content Services Certified Administrator (ACSCA), Alfresco Content Services Certified Engineer (ACSCE) as well as OpenText Documentum Certified Administrator. His industry experience spans the Public Sector, IT Services, Financial Services\/Banking, and the Pharmaceutical industry.","sameAs":["https:\/\/blog.dbi-services.com\/author\/morgan-patou\/","https:\/\/x.com\/MorganPatou"],"url":"https:\/\/www.dbi-services.com\/blog\/author\/morgan-patou\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/17939","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/20"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=17939"}],"version-history":[{"count":8,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/17939\/revisions"}],"predecessor-version":[{"id":17953,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/17939\/revisions\/17953"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=17939"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=17939"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=17939"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=17939"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}