Recently at a customer, I faced a new issue I didn’t see before on the xPlore Dsearch component. Basically, the processes appeared to be starting and the “dsearchadmin.war” was deployed properly but it always failed for the “dsearch.war” upon startup of WildFly. However, it was possible to “deploy” it once WildFly was running… This issue wasn’t too complex to debug because some interesting hints were displayed on the logs but it’s an interesting behavior because you could think that everything is running while it’s not. The initial issue was the following one:

[xplore@ds-0 ~]$ cd $JBOSS_HOME/server
[xplore@ds-0 server]$
[xplore@ds-0 server]$ $STARTSTOP start
  **
  **  The PrimaryDsearch is shutdown
  **
INFO - Starting the PrimaryDsearch...
  **
  **  The PrimaryDsearch is running with PID: 9178
  **
[xplore@ds-0 server]$
[xplore@ds-0 server]$ ps uxf
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
xplore    9075  0.0  0.0  13428  1552 pts/1    S    12:06   0:00 /bin/sh ./startPrimaryDsearch.sh
xplore    9077  0.0  0.0  13448  1744 pts/1    S    12:06   0:00  \_ /bin/sh $XPLORE_HOME/wildfly17.0.1/bin/standalone.sh
xplore    9178 83.2  1.1 10193500 1580936 pts/1 Sl  12:06   1:32      \_ $XPLORE_HOME/java64/JAVA_LINK/bin/java -D[Standalone] -server -Xms8g -Xmx8g -XX:MaxMetaspaceSize=512m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:G1HeapRegionSize=4m -Djbo
xplore    1401  0.0  0.0  13568  2260 pts/2    Ss   11:50   0:00 bash -l
xplore    9940  0.0  0.0  53336  1864 pts/2    R+   12:07   0:00  \_ ps uxf
[xplore@ds-0 server]$
[xplore@ds-0 server]$ cat DctmServer_PrimaryDsearch/log/server.log
...
...
2022-07-13 12:06:30,311 UTC INFO  [org.jboss.ws.cxf.deployment] (MSC service thread 1-2) JBWS024074: WSDL published to: file:$XPLORE_HOME/wildfly17.0.1/server/DctmServer_PrimaryDsearch/data/wsdl/dsearch.war/ESSAdminWebServiceService.wsdl
2022-07-13 12:06:30,356 UTC WARN  [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0274: Excluded dependency org.slf4j.impl via jboss-deployment-structure.xml does not exist.
2022-07-13 12:06:30,356 UTC INFO  [org.jboss.as.webservices] (MSC service thread 1-1) WFLYWS0003: Starting service jboss.ws.endpoint."dsearch.war"."com.emc.documentum.core.fulltext.indexserver.admin.controller.ESSAdminWebService"
2022-07-13 12:06:30,421 UTC WARN  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0101: Duplicate servlet mapping /ESSAdminWebService found
2022-07-13 12:06:34,847 UTC INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 67) WFLYUT0021: Registered web context: '/dsearchadmin' for server 'default-server'
2022-07-13 12:06:34,996 UTC INFO  [javax.enterprise.resource.webcontainer.jsf.config] (ServerService Thread Pool -- 74) Initializing Mojarra 2.3.9.SP02 for context '/dsearch'
2022-07-13 12:06:40,012 UTC ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 74) MSC000001: Failed to start service jboss.deployment.unit."dsearch.war".undertow-deployment: org.jboss.msc.service.StartException in service jboss.deployment.unit."dsearch.war".undertow-deployment: java.lang.ExceptionInInitializerError
        at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at [email protected]//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at [email protected]//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
        at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
        at java.base/java.lang.Thread.run(Thread.java:829)
        at [email protected]//org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: java.lang.ExceptionInInitializerError
        at deployment.dsearch.war//com.emc.cma.cps.management.CPSManager.shutdownManager(CPSManager.java:806)
        at deployment.dsearch.war//com.emc.cma.cps.management.CPSManager.terminate(CPSManager.java:160)
        at deployment.dsearch.war//com.emc.cma.cps.management.CPSManager.init(CPSManager.java:1019)
        at deployment.dsearch.war//com.emc.cma.cps.management.CPSManager.getInstance(CPSManager.java:114)
        at deployment.dsearch.war//com.emc.cma.cps.services.CPSServiceImpl.getVersion(CPSServiceImpl.java:46)
        at deployment.dsearch.war//com.emc.cma.cps.rt.ContentProcessingServiceLocalStub.getVersion(ContentProcessingServiceLocalStub.java:56)
        at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.cps.CPSSubmitter.connectCPS(CPSSubmitter.java:301)
        at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.cps.CPSSubmitter.<init>(CPSSubmitter.java:159)
        at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.cps.CPSRouter.addCPS(CPSRouter.java:108)
        at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.cps.CPSRouter.<init>(CPSRouter.java:85)
        at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.core.ESSContext.initialize(ESSContext.java:240)
        at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.core.ESSNode.startUp(ESSNode.java:67)
        at deployment.dsearch.war//com.emc.documentum.core.fulltext.webapp.IndexServerServlet.init(IndexServerServlet.java:48)
        at [email protected]//io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)
        at [email protected]//org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78)
        at [email protected]//io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)
        at [email protected]//io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:303)
        at [email protected]//io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:143)
        at [email protected]//io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:583)
        at [email protected]//io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:554)
        at [email protected]//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
        at [email protected]//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
        at [email protected]//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
        at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
        at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
        at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
        at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
        at [email protected]//io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:596)
        at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:97)
        at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:78)
        ... 8 more
Caused by: java.lang.NullPointerException
        at java.base/sun.nio.fs.UnixPath.normalizeAndCheck(UnixPath.java:75)
        at java.base/sun.nio.fs.UnixPath.<init>(UnixPath.java:69)
        at java.base/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:279)
        at java.base/java.nio.file.Path.of(Path.java:147)
        at java.base/java.nio.file.Paths.get(Paths.java:69)
        at deployment.dsearch.war//com.emc.cma.cps.util.FileUtils.isAbsoluteDirectory(FileUtils.java:8)
        at deployment.dsearch.war//com.emc.cma.cps.management.CPSConfiguration.getEffectiveTempDirectory(CPSConfiguration.java:394)
        at deployment.dsearch.war//com.emc.cma.cps.processor.common.CPSContentBufferManager.<clinit>(CPSContentBufferManager.java:166)
        ... 38 more

2022-07-13 12:06:40,020 UTC ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "dsearch.war")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.unit.\"dsearch.war\".undertow-deployment" => "java.lang.ExceptionInInitializerError
    Caused by: java.lang.ExceptionInInitializerError
    Caused by: java.lang.NullPointerException"}}
2022-07-13 12:06:41,370 UTC INFO  [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0010: Deployed "dsearchadmin.war" (runtime-name : "dsearchadmin.war")
2022-07-13 12:06:41,370 UTC INFO  [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0010: Deployed "dsearch.war" (runtime-name : "dsearch.war")
2022-07-13 12:06:41,373 UTC INFO  [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
WFLYCTL0186:   Services which failed to start:      service jboss.deployment.unit."dsearch.war".undertow-deployment: java.lang.ExceptionInInitializerError

2022-07-13 12:06:41,462 UTC INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
2022-07-13 12:06:41,468 UTC ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: WildFly Full 17.0.1.Final (WildFly Core 9.0.2.Final) started (with errors) in 36174ms - Started 1074 of 1323 services (2 services failed or missing dependencies, 400 services are lazy, passive or on-demand)
[xplore@ds-0 server]$

As you can see above, there are no cps_daemon child processes for the PrimaryDsearch, which is a good indication that even if the WildFly process is running, the xPlore PrimaryDsearch isn’t fully up&running and that something went wrong. On the logs, and especially on the stack trace above, the issue appears to be on the initialization/startup of the PrimaryDsearch and more specifically around its included CPS. Looking at the URL obviously fails since the application has been switched to failed state so the first thing I tried was simply to test a redeploy of the application. The custom script $STARTSTOP we are using (c.f. above) already take care to force a redeployment, in case one of the Applications is in “undeployed” or “failed” state before WildFly starts, but I still tried to do it manually during runtime, to see if it would work:

[xplore@ds-0 server]$ curl -kI http://`hostname -f`:9300/dsearch
HTTP/1.1 404 Not Found
Connection: keep-alive
Server: FT1
Content-Length: 74
Content-Type: text/html
Date: Wed, 13 Jul 2022 12:07:36 GMT
[xplore@ds-0 server]$
[xplore@ds-0 server]$ cd DctmServer_PrimaryDsearch/deployments/
[xplore@ds-0 deployments]$
[xplore@ds-0 deployments]$ ls -l
total 24
drwxrwx--- 11 xplore xplore 4096 Jul 13 07:17 dsearchadmin.war
-rw-r-----  1 xplore xplore   16 Jul 13 07:17 dsearchadmin.war.deployed
drwxrwx---  4 xplore xplore 4096 Jul 13 07:17 dsearch.war
-rw-r-----  1 xplore xplore  251 Jul 13 12:06 dsearch.war.failed
-rw-rw----  1 xplore xplore 9078 Jul 13 07:17 README.txt
[xplore@ds-0 deployments]$
[xplore@ds-0 deployments]$ mv dsearch.war.failed dsearch.war.dodeploy
[xplore@ds-0 deployments]$
[xplore@ds-0 deployments]$ ls -l
total 24
drwxrwx--- 11 xplore xplore 4096 Jul 13 07:17 dsearchadmin.war
-rw-r-----  1 xplore xplore   16 Jul 13 07:17 dsearchadmin.war.deployed
drwxrwx---  4 xplore xplore 4096 Jul 13 07:17 dsearch.war
-rw-r-----  1 xplore xplore  251 Jul 13 12:06 dsearch.war.dodeploy
-rw-r-----  1 xplore xplore   11 Jul 13 12:08 dsearch.war.isdeploying
-rw-rw----  1 xplore xplore 9078 Jul 13 07:17 README.txt
[xplore@ds-0 deployments]$
[xplore@ds-0 deployments]$ sleep 20
[xplore@ds-0 deployments]$
[xplore@ds-0 deployments]$ ls -l
total 20
drwxrwx--- 11 xplore xplore 4096 Jul 13 07:17 dsearchadmin.war
-rw-r-----  1 xplore xplore   16 Jul 13 07:17 dsearchadmin.war.deployed
drwxrwx---  4 xplore xplore 4096 Jul 13 07:17 dsearch.war
-rw-r-----  1 xplore xplore   11 Jul 13 07:18 dsearch.war.deployed
-rw-rw----  1 xplore xplore 9078 Jul 13 07:17 README.txt
[xplore@ds-0 deployments]$

Somehow, it looked like WildFly was able to deploy the dsearch.war application this time… So, all happy, I looked at the server.log again to confirm that:

[xplore@ds-0 deployments]$ cat ../log/server.log
...
2022-07-13 12:06:40,020 UTC ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "dsearch.war")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.unit.\"dsearch.war\".undertow-deployment" => "java.lang.ExceptionInInitializerError
    Caused by: java.lang.ExceptionInInitializerError
    Caused by: java.lang.NullPointerException"}}
2022-07-13 12:06:41,370 UTC INFO  [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0010: Deployed "dsearchadmin.war" (runtime-name : "dsearchadmin.war")
2022-07-13 12:06:41,370 UTC INFO  [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0010: Deployed "dsearch.war" (runtime-name : "dsearch.war")
2022-07-13 12:06:41,373 UTC INFO  [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
WFLYCTL0186:   Services which failed to start:      service jboss.deployment.unit."dsearch.war".undertow-deployment: java.lang.ExceptionInInitializerError

2022-07-13 12:06:41,462 UTC INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
2022-07-13 12:06:41,468 UTC ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: WildFly Full 17.0.1.Final (WildFly Core 9.0.2.Final) started (with errors) in 36174ms - Started 1074 of 1323 services (2 services failed or missing dependencies, 400 services are lazy, passive or on-demand)
2022-07-13 12:08:51,775 UTC INFO  [org.jboss.as.webservices] (MSC service thread 1-5) WFLYWS0004: Stopping service jboss.ws.endpoint."dsearch.war"."com.emc.documentum.core.fulltext.indexserver.admin.controller.ESSAdminWebService"
2022-07-13 12:08:52,508 UTC INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0028: Stopped deployment dsearch.war (runtime-name: dsearch.war) in 739ms
2022-07-13 12:08:52,517 UTC INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0027: Starting deployment of "dsearch.war" (runtime-name: "dsearch.war")
...
...
2022-07-13 12:09:03,839 UTC INFO  [org.jboss.ws.cxf.deployment] (MSC service thread 1-1) JBWS024074: WSDL published to: file:$XPLORE_HOME/wildfly17.0.1/server/DctmServer_PrimaryDsearch/data/wsdl/dsearch.war/ESSAdminWebServiceService.wsdl
2022-07-13 12:09:03,884 UTC INFO  [org.jboss.as.webservices] (MSC service thread 1-1) WFLYWS0003: Starting service jboss.ws.endpoint."dsearch.war"."com.emc.documentum.core.fulltext.indexserver.admin.controller.ESSAdminWebService"
2022-07-13 12:09:03,884 UTC WARN  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0274: Excluded dependency org.slf4j.impl via jboss-deployment-structure.xml does not exist.
2022-07-13 12:09:03,908 UTC WARN  [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0101: Duplicate servlet mapping /ESSAdminWebService found
2022-07-13 12:09:07,181 UTC INFO  [javax.enterprise.resource.webcontainer.jsf.config] (ServerService Thread Pool -- 80) Initializing Mojarra 2.3.9.SP02 for context '/dsearch'
2022-07-13 12:09:10,106 UTC INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 80) WFLYUT0021: Registered web context: '/dsearch' for server 'default-server'
2022-07-13 12:09:10,168 UTC INFO  [org.jboss.as.server] (DeploymentScanner-threads - 2) WFLYSRV0016: Replaced deployment "dsearch.war" with deployment "dsearch.war"
[xplore@ds-0 deployments]$

As you can see above, the initial deployment failed (12:06:40) but then, by manually asking for a redeployment (12:08:51), it looks like WildFly was able to successfully deploy it, no? Let’s look at the URL:

[xplore@ds-0 deployments]$ curl -kI http://`hostname -f`:9300/dsearch
HTTP/1.1 302 Found
Connection: keep-alive
Server: FT1
Location: http://ds-0.domain.com:9300/dsearch/
Content-Length: 0
Date: Wed, 13 Jul 2022 12:10:01 GMT

[xplore@ds-0 deployments]$

The URL is now responding with a correct “302 Found” which means that it looks good, right? To cross-check, I just added the “/” at the end to use the real redirected location:

[xplore@ds-0 deployments]$ curl -kI http://`hostname -f`:9300/dsearch/
HTTP/1.1 404 Not Found
Connection: keep-alive
Server: FT1
Content-Length: 0
Date: Wed, 13 Jul 2022 12:10:05 GMT

[xplore@ds-0 deployments]$

That’s very peculiar, the deployment appears to be successful, and the first URL appears to work as well but if you really look behind the first link, it’s not working. Of course, the cps_daemon child processes of the PrimaryDsearch are also not present. Therefore, you might think that it started but it didn’t. As mentioned previously, the stack trace appeared to be linked to CPS initialization. Therefore, I looked at the CPS configuration file which is, by default, $XPLORE_HOME/dsearch/cps/cps_daemon/###DS_NAME###_local_configuration.xml for a Dsearch or $XPLORE_HOME/dsearch/cps/cps_daemon/###CPS_NAME###_configuration.xml for a CPS Only. In any cases, you can find the correct name/path of this file in the “cps.properties” file (for a CPS Only, it’s inside the cps.war, not inside the dsearch.war):

[xplore@ds-0 deployments]$ cps_conf_file=$(grep cps.configuration.file dsearch.war/WEB-INF/classes/cps.properties | awk -F= '{print $2}')
[xplore@ds-0 deployments]$
[xplore@ds-0 deployments]$ echo ${cps_conf_file}
$XPLORE_HOME/dsearch/cps/cps_daemon/PrimaryDsearch_local_configuration.xml
[xplore@ds-0 deployments]$
[xplore@ds-0 deployments]$ ls -l ${cps_conf_file}
ls: cannot access $XPLORE_HOME/dsearch/cps/cps_daemon/PrimaryDsearch_local_configuration.xml: No such file or directory
[xplore@ds-0 deployments]$

It appears that the CPS configuration file is gone and that is what is causing this issue, since these processes cannot be initialized. This file was unfortunately removed by mistake a few hours earlier and because of that, the xPlore processes couldn’t start anymore. Restoring the file (from the closest backup) and then restarting xPlore processes was sufficient to have the services back online (since the “/dsearch” isn’t really reachable, there are obviously errors during the shutdown):

[xplore@ds-0 deployments]$ $STARTSTOP stop
  **
  **  The PrimaryDsearch is running with PID: 9178
  **
INFO - Stopping the PrimaryDsearch...
Instance {PrimaryDsearch} is about to shut down, wait for shutdown complete message.
Exception in thread "main" java.lang.IllegalArgumentException: Fail to connect remote server ...
{
    "outcome" => "success",
    "result" => undefined
}
  **
  **  The PrimaryDsearch is shutdown
  **
[xplore@ds-0 deployments]$
[xplore@ds-0 deployments]$ # Restoring the file from the closest backup. Post restore:
[xplore@ds-0 deployments]$ ls -l ${cps_conf_file}
-rw-rw---- 1 xplore xplore 9269 Jul 13 12:22 $XPLORE_HOME/dsearch/cps/cps_daemon/PrimaryDsearch_local_configuration.xml
[xplore@ds-0 deployments]$
[xplore@ds-0 deployments]$ $STARTSTOP start
  **
  **  The PrimaryDsearch is shutdown
  **
INFO - Starting the PrimaryDsearch...
  **
  **  The PrimaryDsearch is running with PID: 14826
  **
[xplore@ds-0 deployments]$

And cross-checking with the logs and URLs, it started properly:

[xplore@ds-0 deployments]$ cat ../log/server.log
...
2022-07-13 12:23:08,670 UTC INFO  [org.jboss.ws.cxf.deployment] (MSC service thread 1-2) JBWS024074: WSDL published to: file:$XPLORE_HOME/wildfly17.0.1/server/DctmServer_PrimaryDsearch/data/wsdl/dsearch.war/ESSAdminWebServiceService.wsdl
2022-07-13 12:23:08,714 UTC INFO  [org.jboss.as.webservices] (MSC service thread 1-6) WFLYWS0003: Starting service jboss.ws.endpoint."dsearch.war"."com.emc.documentum.core.fulltext.indexserver.admin.controller.ESSAdminWebService"
2022-07-13 12:23:08,715 UTC WARN  [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0274: Excluded dependency org.slf4j.impl via jboss-deployment-structure.xml does not exist.
2022-07-13 12:23:08,755 UTC WARN  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0101: Duplicate servlet mapping /ESSAdminWebService found
2022-07-13 12:23:11,846 UTC INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 76) WFLYUT0021: Registered web context: '/dsearchadmin' for server 'default-server'
2022-07-13 12:23:13,634 UTC INFO  [javax.enterprise.resource.webcontainer.jsf.config] (ServerService Thread Pool -- 93) Initializing Mojarra 2.3.9.SP02 for context '/dsearch'
2022-07-13 12:23:27,323 UTC INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 93) WFLYUT0021: Registered web context: '/dsearch' for server 'default-server'
2022-07-13 12:23:27,356 UTC INFO  [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0010: Deployed "dsearch.war" (runtime-name : "dsearch.war")
2022-07-13 12:23:27,357 UTC INFO  [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0010: Deployed "dsearchadmin.war" (runtime-name : "dsearchadmin.war")
2022-07-13 12:23:27,607 UTC INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
2022-07-13 12:23:27,614 UTC INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 17.0.1.Final (WildFly Core 9.0.2.Final) started in 44869ms - Started 1077 of 1323 services (400 services are lazy, passive or on-demand)
[xplore@ds-0 deployments]$
[xplore@ds-0 deployments]$ curl -kI http://`hostname -f`:9300/dsearch
HTTP/1.1 302 Found
Connection: keep-alive
Server: FT1
Location: http://ds-0.domain.com:9300/dsearch/
Content-Length: 0
Date: Wed, 13 Jul 2022 12:26:05 GMT

[xplore@ds-0 deployments]$
[xplore@ds-0 deployments]$ curl -kI http://`hostname -f`:9300/dsearch/
HTTP/1.1 259 Unknown
Connection: keep-alive
Server: FT1
Content-Type: text/html;charset=UTF-8
Content-Length: 66
Date: Wed, 13 Jul 2022 12:26:07 GMT

[xplore@ds-0 deployments]$

The HTTP response code of 259 is normal for the dsearch URL so it shows that it works properly now. All cps_daemon child processes are also present and there are no issues on any logs. So just to conclude on this blog, it’s not because it appears that WildFly properly deployed the xPlore applications that they are really working! Make sure to always check the URLs and processes, that’s always a better indication. The best is of course to test xPlore end-to-end from Repository Searches & Indexing but that can take some time if it’s not automated/scripted.