Documentum is an inexhaustible source of issues and that’s great because I can continuously write blogs on it, and I always have materials for that. In this blog, I will talk about the ACS not being able to start on the MethodServer because of an ActiveMQ error. As you might know, WildFly has been coming with an integrated ActiveMQ for quite some time now and therefore, in case something goes wrong with ActiveMQ, it might impact your Documentum applications deployed on WildFly.
The issue I will talk about today happened after a simple restart of the MethodServer that has been done simply because some Documentum jars were updated. Unfortunately, the startup failed with the following errors:
[dmadmin@cs-0 ~]$ cd $JMS_HOME/server/DctmServer_MethodServer/log
[dmadmin@cs-0 log]$ cat server.log
...
2022-05-02 06:54:23,181 UTC INFO [org.jboss.modules] (main) JBoss Modules version 1.9.1.Final
2022-05-02 06:54:23,635 UTC INFO [org.jboss.msc] (main) JBoss MSC version 1.4.8.Final
2022-05-02 06:54:23,644 UTC INFO [org.jboss.threads] (main) JBoss Threads version 2.3.3.Final
2022-05-02 06:54:23,801 UTC INFO [org.jboss.as] (MSC service thread 1-1) WFLYSRV0049: WildFly Full 17.0.1.Final (WildFly Core 9.0.2.Final) starting
2022-05-02 06:54:24,701 UTC INFO [org.wildfly.security] (ServerService Thread Pool -- 25) ELY00001: WildFly Elytron version 1.9.1.Final
...
2022-05-02 06:54:31,365 UTC INFO [org.jboss.as.patching] (MSC service thread 1-4) WFLYPAT0050: WildFly Full cumulative patch ID is: base, one-off patches include: none
2022-05-02 06:54:31,403 UTC INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-3) WFLYDS0013: Started FileSystemDeploymentService for directory $DOCUMENTUM/wildfly17.0.1/server/DctmServer_MethodServer/deployments
2022-05-02 06:54:31,406 UTC INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0027: Starting deployment of "ServerApps.ear" (runtime-name: "ServerApps.ear")
2022-05-02 06:54:31,406 UTC INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0027: Starting deployment of "error.war" (runtime-name: "error.war")
2022-05-02 06:54:31,406 UTC INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0027: Starting deployment of "acs.ear" (runtime-name: "acs.ear")
2022-05-02 06:54:31,430 UTC INFO [org.jboss.as.remoting] (MSC service thread 1-7) WFLYRMT0001: Listening on 127.0.0.1:9084
2022-05-02 06:54:31,447 UTC ERROR [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 62) AMQ224097: Failed to start server: java.io.IOException: Input/output error
at java.base/sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
at java.base/sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:54)
at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:274)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:233)
at java.base/sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:811)
at java.base/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:796)
at [email protected]//org.apache.activemq.artemis.core.server.NodeManager.createNodeId(NodeManager.java:209)
at [email protected]//org.apache.activemq.artemis.core.server.NodeManager.setUpServerLockFile(NodeManager.java:195)
at [email protected]//org.apache.activemq.artemis.core.server.impl.FileLockNodeManager.start(FileLockNodeManager.java:76)
at [email protected]//org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:576)
at [email protected]//org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:522)
at [email protected]//org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:373)
at org.wildfly.extension.messaging-activemq//org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:206)
at org.wildfly.extension.messaging-activemq//org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65)
at org.wildfly.extension.messaging-activemq//org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at [email protected]//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at [email protected]//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1348)
at java.base/java.lang.Thread.run(Thread.java:829)
at [email protected]//org.jboss.threads.JBossThread.run(JBossThread.java:485)
2022-05-02 06:54:31,561 UTC INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0006: Undertow HTTPS listener default-https listening on 0.0.0.0:9082
2022-05-02 06:54:31,650 UTC INFO [org.jboss.ws.common.management] (MSC service thread 1-7) JBWS022052: Starting JBossWS 5.3.0.Final (Apache CXF 3.3.2)
2022-05-02 06:54:32,837 UTC INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-3) ISPN000128: Infinispan version: Infinispan 'Infinity Minus ONE +2' 9.4.14.Final
2022-05-02 06:54:33,191 UTC INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 62) WFLYCLINF0002: Started client-mappings cache from ejb container
2022-05-02 06:54:33,317 UTC INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 63) WFLYUT0021: Registered web context: '/' for server 'default-server'
...
2022-05-02 06:55:48,364 UTC INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0207: Starting subdeployment (runtime-name: "bocs.war")
2022-05-02 06:55:48,364 UTC INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0207: Starting subdeployment (runtime-name: "documentum-bocs-ws.war")
2022-05-02 06:55:49,289 UTC WARN [org.jboss.as.dependency.private] (MSC service thread 1-3) WFLYSRV0018: Deployment "deployment.acs.ear" is using a private module ("org.jboss.as.jmx") which may be changed or removed in future versions without notice.
...
2022-05-02 06:55:50,516 UTC INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 70) WFLYUT0021: Registered web context: '/bocs-ws' for server 'default-server'
2022-05-02 06:55:58,622 UTC INFO [org.jboss.as.server] (Thread-1) WFLYSRV0220: Server shutdown has been requested via an OS signal
2022-05-02 06:55:58,656 UTC INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-7) WFLYJCA0019: Stopped Driver service with driver-name = h2
2022-05-02 06:55:58,666 UTC INFO [org.jboss.as.mail.extension] (MSC service thread 1-8) WFLYMAIL0002: Unbound mail session [java:jboss/mail/Default]
2022-05-02 06:55:58,675 UTC INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 71) WFLYUT0022: Unregistered web context: '/bocs-ws' from server 'default-server'
2022-05-02 06:55:58,675 UTC INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 69) WFLYUT0022: Unregistered web context: '/' from server 'default-server'
2022-05-02 06:55:58,691 UTC ERROR [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 70) WFLYMSGAMQ0003: Exception while stopping JMS server: java.lang.NullPointerException
at [email protected]//org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.freezeConnections(ActiveMQServerImpl.java:1302)
at [email protected]//org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1104)
at [email protected]//org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1027)
at [email protected]//org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:872)
at [email protected]//org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:866)
at [email protected]//org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.stop(JMSServerManagerImpl.java:393)
at org.wildfly.extension.messaging-activemq//org.wildfly.extension.messaging.activemq.jms.JMSService.doStop(JMSService.java:218)
at org.wildfly.extension.messaging-activemq//org.wildfly.extension.messaging.activemq.jms.JMSService.access$100(JMSService.java:65)
at org.wildfly.extension.messaging-activemq//org.wildfly.extension.messaging.activemq.jms.JMSService$2.run(JMSService.java:122)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at [email protected]//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at [email protected]//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at java.base/java.lang.Thread.run(Thread.java:829)
at [email protected]//org.jboss.threads.JBossThread.run(JBossThread.java:485)
2022-05-02 06:55:58,700 UTC INFO [javax.enterprise.resource.webservices.jaxws.servlet.http] (ServerService Thread Pool -- 71) WSSERVLET15: JAX-WS servlet destroyed
2022-05-02 06:55:58,703 UTC INFO [javax.enterprise.resource.webservices.jaxws.server.http] (ServerService Thread Pool -- 71) WSSERVLET13: JAX-WS context listener destroyed
2022-05-02 06:55:59,288 UTC INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0028: Stopped deployment error.war (runtime-name: error.war) in 645ms
...
2022-05-02 06:55:59,502 UTC INFO [stdout] (FelixStartLevel) ERROR: Error starting file:$DOCUMENTUM/wildfly17.0.1/server/DctmServer_MethodServer/deployments/acs.ear/lib/Web.jar (org.osgi.framework.BundleException: Activator start error in bundle web [4].)
2022-05-02 06:55:59,502 UTC ERROR [stderr] (FelixStartLevel) java.lang.ExceptionInInitializerError
2022-05-02 06:55:59,503 UTC ERROR [stderr] (FelixStartLevel) at com.documentum.acs.sdi.servlet.AcsHelper.<clinit>(AcsHelper.java:161)
2022-05-02 06:55:59,503 UTC ERROR [stderr] (FelixStartLevel) at com.documentum.acs.sdi.servlet.ACS.<clinit>(ACS.java:404)
2022-05-02 06:55:59,503 UTC ERROR [stderr] (FelixStartLevel) at com.documentum.acs.sdi.osgi.ServletActivator$HttpTrackerCustomizer.addingService(ServletActivator.java:48)
2022-05-02 06:55:59,503 UTC ERROR [stderr] (FelixStartLevel) at deployment.acs.ear//org.osgi.util.tracker.ServiceTracker$Tracked.trackAdding(ServiceTracker.java:1030)
2022-05-02 06:55:59,503 UTC ERROR [stderr] (FelixStartLevel) at deployment.acs.ear//org.osgi.util.tracker.ServiceTracker$Tracked.trackInitialServices(ServiceTracker.java:891)
2022-05-02 06:55:59,504 UTC ERROR [stderr] (FelixStartLevel) at deployment.acs.ear//org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:296)
2022-05-02 06:55:59,504 UTC ERROR [stderr] (FelixStartLevel) at deployment.acs.ear//org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:235)
2022-05-02 06:55:59,504 UTC ERROR [stderr] (FelixStartLevel) at com.documentum.acs.sdi.osgi.ServletActivator.start(ServletActivator.java:25)
2022-05-02 06:55:59,504 UTC ERROR [stderr] (FelixStartLevel) at deployment.acs.ear//org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:589)
2022-05-02 06:55:59,504 UTC ERROR [stderr] (FelixStartLevel) at deployment.acs.ear//org.apache.felix.framework.Felix._startBundle(Felix.java:1671)
2022-05-02 06:55:59,505 UTC ERROR [stderr] (FelixStartLevel) at deployment.acs.ear//org.apache.felix.framework.Felix.startBundle(Felix.java:1588)
2022-05-02 06:55:59,505 UTC ERROR [stderr] (FelixStartLevel) at deployment.acs.ear//org.apache.felix.framework.Felix.setFrameworkStartLevel(Felix.java:1180)
2022-05-02 06:55:59,505 UTC ERROR [stderr] (FelixStartLevel) at deployment.acs.ear//org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:265)
2022-05-02 06:55:59,505 UTC ERROR [stderr] (FelixStartLevel) at java.base/java.lang.Thread.run(Thread.java:829)
2022-05-02 06:55:59,513 UTC ERROR [stderr] (FelixStartLevel) Caused by: com.google.inject.CreationException: Guice creation errors:
...
2022-05-02 06:55:59,677 UTC INFO [org.wildfly.naming] (FelixStartLevel) WildFly Naming version 1.0.10.Final
2022-05-02 06:55:59,838 UTC INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 68) WFLYUT0021: Registered web context: '/ACS' for server 'default-server'
2022-05-02 06:55:59,838 UTC INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 68) WFLYUT0022: Unregistered web context: '/ACS' from server 'default-server'
2022-05-02 06:55:59,908 UTC INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0019: Host default-host stopping
2022-05-02 06:55:59,925 UTC INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0208: Stopped subdeployment (runtime-name: documentum-bocs-ws.war) in 1285ms
2022-05-02 06:55:59,928 UTC INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0008: Undertow HTTPS listener default-https suspending
2022-05-02 06:55:59,928 UTC INFO [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0008: Undertow HTTP listener default suspending
2022-05-02 06:55:59,929 UTC INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0007: Undertow HTTPS listener default-https stopped, was bound to 0.0.0.0:9082
2022-05-02 06:55:59,929 UTC INFO [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 0.0.0.0:9080
2022-05-02 06:55:59,930 UTC INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0004: Undertow 2.0.21.Final stopping
2022-05-02 06:55:59,930 UTC INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 68) WFLYCLINF0003: Stopped client-mappings cache from ejb container
2022-05-02 06:56:00,101 UTC INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0208: Stopped subdeployment (runtime-name: bocs.war) in 1457ms
[dmadmin@cs-0 log]$
As you can see on the above log, the issue appears to be on the ACS application (the ServerApps URL responds but not the ACS one) and it’s definitively caused by ActiveMQ that doesn’t start properly: “AMQ224097: Failed to start server: java.io.IOException: Input/output error“. Just to make sure, I checked previous logs but there really was no issues on previous startup as well as during the previous runtime. No errors at all on the JMS log files so the issue really started during this startup. The only thing that changed, and the reason for this restart, as I mentioned previously, was to update a few libraries. However, these jar files were Documentum specific, nothing related to WildFly or ActiveMQ. According to the error stack, it seemed to me like it could be caused by some kind of NFS problem. Maybe a disconnection of the NFS at the wrong moment or something similar that could cause the ActiveMQ to either lose a lock file or its status. So, I looked into the ActiveMQ specific data that WildFly creates when it starts:
[dmadmin@cs-0 log]$ ls -l ../data/
total 0
drwxrwx--- 2 dmadmin dmadmin 152 Jun 12 2021 MethodServer
drwxrwxr-x 5 dmadmin dmadmin 152 Jun 23 2021 activemq
drwxrwxr-x 3 dmadmin dmadmin 152 May 2 07:05 content
drwxrwxr-x 2 dmadmin dmadmin 152 Jun 23 2021 kernel
drwxrwxr-x 2 dmadmin dmadmin 152 Jun 23 2021 timer-service-data
drwxrwxr-x 3 dmadmin dmadmin 152 Jun 23 2021 tx-object-store
[dmadmin@cs-0 log]$
[dmadmin@cs-0 log]$ ls -l ../data/activemq/
total 16
drwxrwxr-x 2 dmadmin dmadmin 8192 Jun 23 2021 bindings
drwxrwxr-x 2 dmadmin dmadmin 8192 Apr 22 18:33 journal
drwxrwxr-x 2 dmadmin dmadmin 152 Jun 23 2021 largemessages
[dmadmin@cs-0 log]$
[dmadmin@cs-0 log]$ ls -l ../data/activemq/*
activemq/bindings:
total 4128
-rw-rw-r-- 1 dmadmin dmadmin 1048576 Apr 22 18:34 activemq-bindings-1.bindings
-rw-rw-r-- 1 dmadmin dmadmin 1048576 Apr 22 18:33 activemq-bindings-2.bindings
-rw-rw-r-- 1 dmadmin dmadmin 1048576 Apr 22 18:33 activemq-jms-1.jms
-rw-rw-r-- 1 dmadmin dmadmin 1048576 Apr 22 18:33 activemq-jms-2.jms
activemq/journal:
total 20536
-rw-rw-r-- 1 dmadmin dmadmin 10485760 Apr 22 18:33 activemq-data-1.amq
-rw-rw-r-- 1 dmadmin dmadmin 10485760 Apr 22 18:33 activemq-data-2.amq
-rw-rw-r-- 1 dmadmin dmadmin 19 Apr 22 18:34 server.lock
activemq/largemessages:
total 0
[dmadmin@cs-0 log]$
All the files appeared to be present, and it looked like nothing changed them in the previous days… However, since I assumed the problem was coming from the ActiveMQ data, I still tried to shutdown WildFly, backup these data and then start it again to let it regenerate the files automatically:
[dmadmin@cs-0 log]$ mv ../data/activemq ../data/activemq.old
[dmadmin@cs-0 log]$
[dmadmin@cs-0 log]$ ls -l ../data/
total 0
drwxrwx--- 2 dmadmin dmadmin 152 Jun 12 2021 MethodServer
drwxrwxr-x 5 dmadmin dmadmin 152 Jun 23 2021 activemq.old
drwxrwxr-x 3 dmadmin dmadmin 152 May 2 07:05 content
drwxrwxr-x 2 dmadmin dmadmin 152 Jun 23 2021 kernel
drwxrwxr-x 2 dmadmin dmadmin 152 Jun 23 2021 timer-service-data
drwxrwxr-x 3 dmadmin dmadmin 152 Jun 23 2021 tx-object-store
[dmadmin@cs-0 log]$
[dmadmin@cs-0 log]$ $JMS_HOME/server/startJMSCustom.sh
Starting the Java MethodServer...
The Java MethodServer has been started.
[dmadmin@cs-0 log]$
[dmadmin@cs-0 log]$ ls -l ../data/
total 0
drwxrwx--- 2 dmadmin dmadmin 152 Jun 12 2021 MethodServer
drwxr-x--- 5 dmadmin dmadmin 152 May 2 07:17 activemq
drwxrwxr-x 5 dmadmin dmadmin 152 Jun 23 2021 activemq.old
drwxrwxr-x 3 dmadmin dmadmin 152 May 2 07:16 content
drwxrwxr-x 2 dmadmin dmadmin 152 Jun 23 2021 kernel
drwxrwxr-x 2 dmadmin dmadmin 152 Jun 23 2021 timer-service-data
drwxrwxr-x 3 dmadmin dmadmin 152 Jun 23 2021 tx-object-store
[dmadmin@cs-0 log]$
[dmadmin@cs-0 log]$ ls -l ../data/activemq/*
DctmServer_MethodServer/data/activemq/bindings:
total 4128
-rw-r----- 1 dmadmin dmadmin 1048576 May 2 07:17 activemq-bindings-1.bindings
-rw-r----- 1 dmadmin dmadmin 1048576 May 2 07:17 activemq-bindings-2.bindings
-rw-r----- 1 dmadmin dmadmin 1048576 May 2 07:17 activemq-jms-1.jms
-rw-r----- 1 dmadmin dmadmin 1048576 May 2 07:17 activemq-jms-2.jms
DctmServer_MethodServer/data/activemq/journal:
total 20536
-rw-r----- 1 dmadmin dmadmin 10485760 May 2 07:17 activemq-data-1.amq
-rw-r----- 1 dmadmin dmadmin 10485760 May 2 07:17 activemq-data-2.amq
-rw-r----- 1 dmadmin dmadmin 19 May 2 07:17 server.lock
DctmServer_MethodServer/data/activemq/largemessages:
total 0
[dmadmin@cs-0 log]$
As you can see above, all the files were properly re-created. Therefore, checking the JMS log file now, as well as its URL:
[dmadmin@cs-0 log]$ cat server.log
...
2022-05-02 07:16:56,318 UTC INFO [org.jboss.modules] (main) JBoss Modules version 1.9.1.Final
2022-05-02 07:16:56,782 UTC INFO [org.jboss.msc] (main) JBoss MSC version 1.4.8.Final
2022-05-02 07:16:56,792 UTC INFO [org.jboss.threads] (main) JBoss Threads version 2.3.3.Final
2022-05-02 07:16:56,958 UTC INFO [org.jboss.as] (MSC service thread 1-1) WFLYSRV0049: WildFly Full 17.0.1.Final (WildFly Core 9.0.2.Final) starting
2022-05-02 07:16:57,870 UTC INFO [org.wildfly.security] (ServerService Thread Pool -- 25) ELY00001: WildFly Elytron version 1.9.1.Final
...
2022-05-02 07:16:59,886 UTC INFO [org.jboss.as.patching] (MSC service thread 1-1) WFLYPAT0050: WildFly Full cumulative patch ID is: base, one-off patches include: none
2022-05-02 07:16:59,911 UTC INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-3) WFLYDS0013: Started FileSystemDeploymentService for directory $DOCUMENTUM/wildfly17.0.1/server/DctmServer_MethodServer/deployments
2022-05-02 07:16:59,917 UTC INFO [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0027: Starting deployment of "ServerApps.ear" (runtime-name: "ServerApps.ear")
2022-05-02 07:16:59,919 UTC INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0027: Starting deployment of "error.war" (runtime-name: "error.war")
2022-05-02 07:16:59,918 UTC INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0027: Starting deployment of "acs.ear" (runtime-name: "acs.ear")
2022-05-02 07:16:59,986 UTC INFO [org.jboss.as.remoting] (MSC service thread 1-1) WFLYRMT0001: Listening on 127.0.0.1:9084
2022-05-02 07:17:00,027 UTC INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 62) AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=false,journalDirectory=$DOCUMENTUM/wildfly17.0.1/server/DctmServer_MethodServer/data/activemq/journal,bindingsDirectory=$DOCUMENTUM/wildfly17.0.1/server/DctmServer_MethodServer/data/activemq/bindings,largeMessagesDirectory=$DOCUMENTUM/wildfly17.0.1/server/DctmServer_MethodServer/data/activemq/largemessages,pagingDirectory=$DOCUMENTUM/wildfly17.0.1/server/DctmServer_MethodServer/data/activemq/paging)
2022-05-02 07:17:00,107 UTC INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 62) AMQ221012: Using AIO Journal
2022-05-02 07:17:00,136 UTC INFO [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0006: Undertow HTTPS listener default-https listening on 0.0.0.0:9082
2022-05-02 07:17:00,213 UTC INFO [org.jboss.ws.common.management] (MSC service thread 1-8) JBWS022052: Starting JBossWS 5.3.0.Final (Apache CXF 3.3.2)
2022-05-02 07:17:00,249 UTC INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 62) AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2022-05-02 07:17:00,250 UTC INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 62) AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
2022-05-02 07:17:00,254 UTC INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 62) AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2022-05-02 07:17:00,255 UTC INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 62) AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
2022-05-02 07:17:00,337 UTC INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 62) AMQ221034: Waiting indefinitely to obtain live lock
2022-05-02 07:17:00,337 UTC INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 62) AMQ221035: Live Server Obtained live lock
2022-05-02 07:17:00,952 UTC INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-8) ISPN000128: Infinispan version: Infinispan 'Infinity Minus ONE +2' 9.4.14.Final
2022-05-02 07:17:00,991 UTC INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-6) WFLYMSGAMQ0016: Registered HTTP upgrade for activemq-remoting protocol handled by http-acceptor acceptor
2022-05-02 07:17:00,991 UTC INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-5) WFLYMSGAMQ0016: Registered HTTP upgrade for activemq-remoting protocol handled by http-acceptor-throughput acceptor
2022-05-02 07:17:00,991 UTC INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-1) WFLYMSGAMQ0016: Registered HTTP upgrade for activemq-remoting protocol handled by http-acceptor acceptor
2022-05-02 07:17:00,991 UTC INFO [org.wildfly.extension.messaging-activemq] (MSC service thread 1-4) WFLYMSGAMQ0016: Registered HTTP upgrade for activemq-remoting protocol handled by http-acceptor-throughput acceptor
2022-05-02 07:17:01,264 UTC INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 62) AMQ221007: Server is now live
2022-05-02 07:17:01,265 UTC INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 62) AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.8.1 [default, nodeID=db920184-c9e7-11ec-8ebb-223c2a795152]
2022-05-02 07:17:01,395 UTC INFO [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0007: Registered connection factory java:/JmsXA
2022-05-02 07:17:01,396 UTC INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 66) WFLYMSGAMQ0002: Bound messaging object to jndi name java:jboss/exported/jms/RemoteConnectionFactory
2022-05-02 07:17:01,401 UTC INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 67) WFLYMSGAMQ0002: Bound messaging object to jndi name java:/TouchRpcQueueConnectionFactory
2022-05-02 07:17:01,410 UTC INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 67) WFLYMSGAMQ0002: Bound messaging object to jndi name java:jboss/exported/jms/TouchRpcQueueConnectionFactory
2022-05-02 07:17:01,411 UTC INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 68) WFLYMSGAMQ0002: Bound messaging object to jndi name java:/ConnectionFactory
2022-05-02 07:17:01,411 UTC INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 63) WFLYCLINF0002: Started client-mappings cache from ejb container
2022-05-02 07:17:01,455 UTC INFO [org.apache.activemq.artemis.ra] (MSC service thread 1-8) AMQ151007: Resource adaptor started
2022-05-02 07:17:01,456 UTC INFO [org.jboss.as.connector.services.resourceadapters.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-8) IJ020002: Deployed: file://RaActivatoractivemq-ra
2022-05-02 07:17:01,467 UTC INFO [org.jboss.as.connector.deployment] (MSC service thread 1-3) WFLYJCA0002: Bound JCA ConnectionFactory [java:/JmsXA]
2022-05-02 07:17:01,467 UTC INFO [org.jboss.as.connector.deployment] (MSC service thread 1-8) WFLYJCA0118: Binding connection factory named java:/JmsXA to alias java:jboss/DefaultJMSConnectionFactory
2022-05-02 07:17:01,583 UTC INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 64) WFLYUT0021: Registered web context: '/' for server 'default-server'
...
2022-05-02 07:17:21,632 UTC INFO [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0010: Deployed "acs.ear" (runtime-name : "acs.ear")
2022-05-02 07:17:21,632 UTC INFO [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0010: Deployed "ServerApps.ear" (runtime-name : "ServerApps.ear")
2022-05-02 07:17:21,633 UTC INFO [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0010: Deployed "error.war" (runtime-name : "error.war")
2022-05-02 07:17:21,758 UTC INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
2022-05-02 07:17:21,762 UTC INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 17.0.1.Final (WildFly Core 9.0.2.Final) started in 25839ms - Started 977 of 1235 services (451 services are lazy, passive or on-demand)
[dmadmin@cs-0 log]$
[dmadmin@cs-0 log]$ curl -kI https://`hostname -f`:9082/DmMethods/servlet/DoMethod
HTTP/1.1 200 OK
Connection: keep-alive
Server: MethodServer
Content-Length: 144
Date: Mon, 2 May 2022 07:18:59 GMT
[dmadmin@cs-0 log]$
[dmadmin@cs-0 log]$ curl -kI https://`hostname -f`:9082/ACS/servlet/ACS
HTTP/1.1 200 OK
Connection: keep-alive
Server: MethodServer
Content-Type: text/plain
Content-Length: 48
Date: Mon, 2 May 2022 07:19:07 GMT
[dmadmin@cs-0 log]$
Therefore, this seems to have fixed properly the issue since now both URLs are responding properly. As far as I know, Documentum isn’t using the ActiveMQ included by WildFly and therefore cleaning the data shouldn’t be a problem. No issues were found after that on this environment and therefore, the issue was closed as is. No real root cause was identified either, so I will most probably continue to believe that it was linked to a network and/or NFS issue… If you are facing the same behavior, don’t hesitate to share!
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/MOP_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/OLS_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/GME_web-min-scaled.jpg)