Have you ever faced an issue where dmqdocbroker, iapi, idql and the likes aren’t able to communicate at all with any Docbroker (connection broker)? Here, I’m not talking about potentially wrong hostname, port or connect modes, which might prevent you to reach a Docbroker if it’s not configured properly because this will still most likely reply to you with an error message… I’m really talking about the utility/binaries that cannot communicate anymore, it’s like all messages are sent to the void and nothing will ever respond (is that a black-hole I’m seeing?)!
Earlier this month, I suddenly had this behavior at one of our customer on two out of dozens of Documentum Servers. Everything seemed to be up&running, all the processes were there:
[dmadmin@cs-0 ~]$ ps -ef UID PID PPID C STIME TTY TIME CMD dmadmin 7005 1 0 14:11 ? 00:00:00 ./dmdocbroker -port 1489 -init_file $DOCUMENTUM/dba/Docbroker.ini dmadmin 7014 1 0 14:11 ? 00:00:00 ./dmdocbroker -port 1487 -init_file $DOCUMENTUM/dba/DocbrokerExt.ini dmadmin 7077 1 0 14:11 ? 00:00:07 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini dmadmin 7087 1 0 14:11 ? 00:00:07 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini dmadmin 7100 7077 0 14:11 ? 00:00:00 $DM_HOME/bin/mthdsvr master 0xfd7308a8, 0x7f9f93d81000, 0x223000 1000726 5 7077 GR_REPO $DOCUMENTUM/dba/log dmadmin 7101 7100 0 14:11 ? 00:00:04 $DM_HOME/bin/mthdsvr worker 0xfd7308a8, 0x7f9f93d81000, 0x223000 1000726 5 0 GR_REPO $DOCUMENTUM/dba/log dmadmin 7102 7087 0 14:11 ? 00:00:00 $DM_HOME/bin/mthdsvr master 0xfd7308be, 0x7fe2fe3ac000, 0x223000 1000727 5 7087 REPO1 $DOCUMENTUM/dba/log dmadmin 7121 7102 0 14:11 ? 00:00:03 $DM_HOME/bin/mthdsvr worker 0xfd7308be, 0x7fe2fe3ac000, 0x223000 1000727 5 0 REPO1 $DOCUMENTUM/dba/log dmadmin 7122 7100 0 14:11 ? 00:00:03 $DM_HOME/bin/mthdsvr worker 0xfd7308a8, 0x7f9f93d81000, 0x223000 1000726 5 1 GR_REPO $DOCUMENTUM/dba/log dmadmin 7123 7077 0 14:11 ? 00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini dmadmin 7124 7077 0 14:11 ? 00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini dmadmin 7144 7102 0 14:11 ? 00:00:04 $DM_HOME/bin/mthdsvr worker 0xfd7308be, 0x7fe2fe3ac000, 0x223000 1000727 5 1 REPO1 $DOCUMENTUM/dba/log dmadmin 7148 7087 0 14:11 ? 00:00:00 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini dmadmin 7149 7087 0 14:11 ? 00:00:00 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini dmadmin 7165 7100 0 14:11 ? 00:00:04 $DM_HOME/bin/mthdsvr worker 0xfd7308a8, 0x7f9f93d81000, 0x223000 1000726 5 2 GR_REPO $DOCUMENTUM/dba/log dmadmin 7166 7077 0 14:11 ? 00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini dmadmin 7167 7102 0 14:11 ? 00:00:03 $DM_HOME/bin/mthdsvr worker 0xfd7308be, 0x7fe2fe3ac000, 0x223000 1000727 5 2 REPO1 $DOCUMENTUM/dba/log dmadmin 7168 7087 0 14:11 ? 00:00:00 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini dmadmin 7169 7100 0 14:11 ? 00:00:04 $DM_HOME/bin/mthdsvr worker 0xfd7308a8, 0x7f9f93d81000, 0x223000 1000726 5 3 GR_REPO $DOCUMENTUM/dba/log dmadmin 7187 7077 0 14:11 ? 00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini dmadmin 7190 7102 0 14:11 ? 00:00:03 $DM_HOME/bin/mthdsvr worker 0xfd7308be, 0x7fe2fe3ac000, 0x223000 1000727 5 3 REPO1 $DOCUMENTUM/dba/log dmadmin 7194 7087 0 14:11 ? 00:00:00 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini dmadmin 7210 7100 0 14:11 ? 00:00:03 $DM_HOME/bin/mthdsvr worker 0xfd7308a8, 0x7f9f93d81000, 0x223000 1000726 5 4 GR_REPO $DOCUMENTUM/dba/log dmadmin 7213 7077 0 14:11 ? 00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini dmadmin 7215 7102 0 14:11 ? 00:00:04 $DM_HOME/bin/mthdsvr worker 0xfd7308be, 0x7fe2fe3ac000, 0x223000 1000727 5 4 REPO1 $DOCUMENTUM/dba/log dmadmin 7225 7087 0 14:11 ? 00:00:00 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini dmadmin 7334 1 0 14:12 ? 00:00:00 /bin/sh $JMS_HOME/server/startMethodServer.sh dmadmin 7336 7334 0 14:12 ? 00:00:00 /bin/sh $JMS_HOME/bin/standalone.sh dmadmin 7447 7336 21 14:12 ? 00:02:57 $JAVA_HOME/bin/java -D[Standalone] -server -XX:+UseCompressedOops -server -XX:+UseCompressedOops -Xms8g -Xmx8g -XX:MaxMetaspaceSize=512m -XX dmadmin 7695 7077 0 14:12 ? 00:00:04 ./dm_agent_exec -enable_ha_setup 1 -docbase_name GR_REPO.GR_REPO -docbase_owner dmadmin -sleep_duration 0 dmadmin 7698 7087 0 14:12 ? 00:00:04 ./dm_agent_exec -enable_ha_setup 1 -docbase_name REPO1.REPO1 -docbase_owner dmadmin -sleep_duration 0 dmadmin 7908 7077 0 14:13 ? 00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini dmadmin 7918 7087 0 14:13 ? 00:00:00 ./documentum -docbase_name REPO1 -security acl -init_file $DOCUMENTUM/dba/config/REPO1/server.ini dmadmin 8269 7077 0 14:21 ? 00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini dmadmin 8270 7077 0 14:21 ? 00:00:00 ./documentum -docbase_name GR_REPO -security acl -init_file $DOCUMENTUM/dba/config/GR_REPO/server.ini dmadmin 8327 6370 0 14:27 pts/1 00:00:00 ps -ef [dmadmin@cs-0 ~]$
However, I could see the communication issues by looking at the Repository log because it would show that the AgentExec was actually not connected, even after almost 20 minutes:
[dmadmin@cs-0 ~]$ cd $DOCUMENTUM/dba/log/REPO1/agentexec/ [dmadmin@cs-0 agentexec]$ date Mon Apr 12 14:29:03 UTC 2021 [dmadmin@cs-0 agentexec]$ [dmadmin@cs-0 agentexec]$ tail -8 ../../REPO1.log 2021-04-12T14:11:56.453407 7087[7087] 0000000000000000 [DM_WORKFLOW_I_AGENT_START]info: "Workflow agent worker (pid : 7194, session 010f12345000000c) is started sucessfully." 2021-04-12T14:11:57.455899 7087[7087] 0000000000000000 [DM_SERVER_I_START]info: "Sending Initial Docbroker check-point " 2021-04-12T14:11:57.547764 7087[7087] 0000000000000000 [DM_MQ_I_DAEMON_START]info: "Message queue daemon (pid : 7225, session 010f123450000456) is started sucessfully." 2021-04-12T14:11:58.348442 7223[7223] 010f123450000003 [DM_DOCBROKER_I_PROJECTING]info: "Sending information to Docbroker located on host (cs-0.domain.com) with port (1490). Information: (Config(REPO1), Proximity(1), Status(Open), Dormancy Status(Active))." 2021-04-12T14:11:58.661666 7223[7223] 010f123450000003 [DM_DOCBROKER_I_PROJECTING]info: "Sending information to Docbroker located on host (cs-0.domain.com) with port (1488). Information: (Config(REPO1), Proximity(1), Status(Open), Dormancy Status(Active))." 2021-04-12T14:11:58.959490 7223[7223] 010f123450000003 [DM_DOCBROKER_I_PROJECTING]info: "Sending information to Docbroker located on host (cs-1.domain.com) with port (1490). Information: (Config(REPO1), Proximity(2), Status(Open), Dormancy Status(Active))." Mon Apr 12 14:12:55 2021 [INFORMATION] [AGENTEXEC 7698] Detected during program initialization: Version: 16.4.0200.0256 Linux64 [dmadmin@cs-0 agentexec]$ [dmadmin@cs-0 agentexec]$ # Previous startup from the AgentExec logs showing that it didn't start yet [dmadmin@cs-0 agentexec]$ tail -2 agentexec.log Sat Apr 10 19:20:30 2021 [INFORMATION] [LAUNCHER 23135] Detected during program initialization: Version: 16.4.0200.0256 Linux64 Sun Apr 11 19:20:26 2021 [INFORMATION] [LAUNCHER 2890] Detected during program initialization: Version: 16.4.0200.0256 Linux64 [dmadmin@cs-0 agentexec]$
The interesting part is that the Repositories have all been started properly and projected to the Docbroker. However, any client from the Documentum Server locally wouldn’t be able to connect to the Docbroker. Even more interesting, this was actually a HA environment with 2 CS. The Documentum Server hosting the Primary CS (I will call it cs-0) had the issue while the Documentum Server hosting the Remote CS (I will call it cs-1) had no problem. Executing the dmqdocbroker on the cs-0 to ping the Docbroker of the cs-0 never gave a response, however doing the exact same command to ping the Docbroker of the cs-0 but from the cs-1 host did work without any problem and showed the correct projection of the repositories:
## on cs-0 (Documentum Server hosting the PCS) [dmadmin@cs-0 ~]$ hostname -f cs-0.domain.com [dmadmin@cs-0 ~]$ [dmadmin@cs-0 ~]$ time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping ^C real 0m53.513s user 0m6.132s sys 0m0.578s [dmadmin@cs-0 ~]$ [dmadmin@cs-0 ~]$ time echo quit | iapi REPO1.REPO1 -Udmadmin -Pxxx ^C real 0m46.431s user 0m6.241s sys 0m0.575s [dmadmin@cs-0 ~]$ [dmadmin@cs-0 ~]$ time echo quit | iapi REPO1.cs-1_REPO1 -Udmadmin -Pxxx ^C real 0m35.694s user 0m6.163s sys 0m0.582s [dmadmin@cs-0 ~]$ ## on cs-1 (Documentum Server hosting the RCS) [dmadmin@cs-1 ~]$ hostname -f cs-1.domain.com [dmadmin@cs-1 ~]$ [dmadmin@cs-1 ~]$ time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping dmqdocbroker: A DocBroker Query Tool dmqdocbroker: Documentum Client Library Version: 16.4.0200.0080 Using specified port: 1489 Successful reply from docbroker at host (cs-0) on port(1490) running software version (16.4.0200.0256 Linux64). real 0m3.499s user 0m6.950s sys 0m0.469s [dmadmin@cs-1 ~]$ [dmadmin@cs-1 ~]$ time echo quit | iapi REPO1.cs-1_REPO1 -Udmadmin -Pxxx OpenText Documentum iapi - Interactive API interface Copyright (c) 2018. OpenText Corporation All rights reserved. Client Library Release 16.4.0200.0080 Connecting to Server using docbase REPO1.cs-1_REPO1 [DM_SESSION_I_SESSION_START]info: "Session 010f1234501b635c started for user dmadmin." Connected to OpenText Documentum Server running Release 16.4.0200.0256 Linux64.Oracle Session id is s0 API> Bye real 0m5.032s user 0m7.401s sys 0m0.487s [dmadmin@cs-1 ~]$ [dmadmin@cs-1 ~]$ time echo quit | iapi REPO1.REPO1 -Udmadmin -Pxxx OpenText Documentum iapi - Interactive API interface Copyright (c) 2018. OpenText Corporation All rights reserved. Client Library Release 16.4.0200.0080 Connecting to Server using docbase REPO1.REPO1 [DM_SESSION_I_SESSION_START]info: "Session 010f1234501b6506 started for user dmadmin." Connected to OpenText Documentum Server running Release 16.4.0200.0256 Linux64.Oracle Session id is s0 API> Bye real 0m5.315s user 0m7.976s sys 0m0.515s [dmadmin@cs-1 ~]$
This shows that the issue isn’t the Docbroker or the Repositories themselves but rather the utility/binaries present on the cs-0 that cannot open communication channels with the local Docbroker, for some reasons… Even after setting debugging on the Docbroker, I could see communications when the dmqdocbroker utility was used on the cs-1 host but nothing was showing-up if the same command was used on the cs-0 host instead. You can enable some logs for the Docbroker by adding “trace=true” into the Docbroker.ini file and you can also add some other traces by setting the following environment variables (value can be 1 or 10 for example) and then restart the Docbroker: “export DM_DOCBROKER_TRACE=1; export DM_DEBUG_BROKER=1; export DM_TRANS_LOG=1“. Additionally, you can also add options to the launch script, just like for the Repository part: “-odocbroker_trace -onettrace_all_option -oxxx“.
Unfortunately, the dmqdocbroker utility uses the dmawk binary and the iapi/idql are also binaries so it’s rather difficult to debug further without the source code… After some testing/debugging, I found something rather hard to believe… All the binaries of the Documentum Server looked OK, they were no changes done in the past few weeks and the files were the same (same hash) than on the cs-1 for example. As you probably know, dmqdocbroker/iapi/idql will use the dfc.properties from the folder “$DOCUMENTUM_SHARED/config/” (with $DOCUMENTUM_SHARED=$DOCUMENTUM forced, starting in 16.4). Therefore, I have been looking into this folder for anything that might disrupt the proper behavior of the utility/binaries. All the files in this folder were 100% identical between cs-0 and cs-1, except for the encrypted password of the dm_bof_registry as well as the dfc.keystore since both of these are generated once. This would mean that the issue wouldn’t be there, but it was. I started looking into other areas to try to find the root cause but nothing was working. Then, I came back to the config folder and simply tried to empty it… Somehow, the dmqdocbroker was working again, magically! I mean, it printed many errors because the files log4j.properties, dfc.properties and dfc.keystore weren’t there but it replied something… What to do then? Well, I went step by step, putting back the files one by one, as they are supposed to be, and then executing the dmqdocbroker again to see if it stops working.
The files dfc.properties, log4j.properties, dfcfull.properties, dfc.keystore and all the cache folders were restored properly and the dmqdocbroker was still working without any problem… So what the hell? That’s more or less all of the files, isn’t it? True, that’s all the files, minus the dbor ones: dbor.properties and dbor.properties.lck. At this customer, these files are empty because no configuration was needed. It would be very hard to believe that this could be the issue, right? Well, have a look for yourself:
[dmadmin@cs-0 ~]$ cd $DOCUMENTUM_SHARED/config/
[dmadmin@cs-0 config]$ ls -l
total 140
drwxr-x--- 7 dmadmin dmadmin 4096 Jul 26 2020 ServerApps
drwxr-x--- 9 dmadmin dmadmin 4096 Jul 26 2020 Shared
drwxr-x--- 7 dmadmin dmadmin 4096 Jul 26 2020 acs
drwxr-x--- 7 dmadmin dmadmin 4096 Jul 26 2020 bpm
-rwxr-x--- 1 dmadmin dmadmin 0 Jul 22 2020 dbor.properties
-rw-rw-r-- 1 dmadmin dmadmin 0 Jul 26 2020 dbor.properties.lck
-rw-rw-r-- 1 dmadmin dmadmin 2152 Jul 26 2020 dfc.keystore
-rw-rw-r-- 1 dmadmin dmadmin 481 Jul 26 2020 dfc.properties
-rw-rw---- 1 dmadmin dmadmin 70 Jul 22 2020 dfc.properties.bak.0
-rwxr-x--- 1 dmadmin dmadmin 242 Jul 26 2020 dfc.properties.bak.1
-rw-rw-r-- 1 dmadmin dmadmin 271 Jul 26 2020 dfc.properties.bak.2
-rw-rw-r-- 1 dmadmin dmadmin 323 Jul 26 2020 dfc.properties.bak.3
-rw-rw-r-- 1 dmadmin dmadmin 481 Jul 26 2020 dfc.properties.bak.4
-rw-rw-r-- 1 dmadmin dmadmin 482 Jul 26 2020 dfc.properties.bak.5
-rwxrwx--- 1 dmadmin dmadmin 79268 Jul 22 2020 dfcfull.properties
-rwxr-x--- 1 dmadmin dmadmin 1242 Jul 26 2020 log4j.properties
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ # With the initial content, dmqdocbroker isn't working
[dmadmin@cs-0 config]$ date; time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping; date
Wed Apr 14 07:43:28 UTC 2021
^C
real 0m22.718s
user 0m6.401s
sys 0m0.853s
Wed Apr 14 07:43:51 UTC 2021
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ mkdir test
[dmadmin@cs-0 config]$ mv * test/
mv: cannot move 'test' to a subdirectory of itself, 'test/test'
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ ls -l
total 4
drwxr-x--- 6 dmadmin dmadmin 4096 Apr 14 07:44 test
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ # With the folder empty, dmqdocbroker is "working" (errors but expected ones)
[dmadmin@cs-0 config]$ date; time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping; date
Wed Apr 14 07:45:15 UTC 2021
0 [main] ERROR com.documentum.fc.common.impl.logging.LoggingConfigurator - Problem locating log4j configuration
1 [main] WARN com.documentum.fc.common.impl.logging.LoggingConfigurator - Using default log4j configuration
3 [main] ERROR com.documentum.fc.common.impl.preferences.PreferencesManager - [DFC_PREFERENCE_LOAD_FAILED] Failed to load persistent preferences from null
java.io.FileNotFoundException: dfc.properties
at com.documentum.fc.common.impl.preferences.PreferencesManager.locateMainPersistentStore(PreferencesManager.java:378)
at com.documentum.fc.common.impl.preferences.PreferencesManager.readPersistentProperties(PreferencesManager.java:329)
at com.documentum.fc.common.impl.preferences.PreferencesManager.<init>(PreferencesManager.java:37)
...
2862 [main] ERROR com.documentum.fc.client.security.impl.IdentityManager - [DFC_SECURITY_IDENTITY_INIT] no identity initialization or incomplete identity initialization
DfException:: THREAD: main; MSG: ; ERRORCODE: ff; NEXT: null
at com.documentum.fc.client.security.impl.JKSKeystoreUtil.creteNewKeystoreFile(JKSKeystoreUtil.java:425)
at com.documentum.fc.client.security.impl.JKSKeystoreUtil.createNewKeystore(JKSKeystoreUtil.java:209)
at com.documentum.fc.client.security.impl.DfcIdentityKeystore.applyDfcInitPolicy(DfcIdentityKeystore.java:95)
...
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0200.0080
Using specified port: 1489
Successful reply from docbroker at host (cs-0) on port(1490) running software version (16.4.0200.0256 Linux64).
real 0m3.763s
user 0m6.265s
sys 0m0.672s
Wed Apr 14 07:45:19 UTC 2021
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ ls -l
total 12
drwxr-x--- 8 dmadmin dmadmin 4096 Apr 14 07:45 documentum
-rw-r----- 1 dmadmin dmadmin 3245 Apr 14 07:45 log4j.log
drwxr-x--- 6 dmadmin dmadmin 4096 Apr 14 07:44 test
-rw-r----- 1 dmadmin dmadmin 0 Apr 14 07:45 trace.log
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ rm -rf documentum/ log4j.log trace.log
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ mv test/log4j.properties ./
[dmadmin@cs-0 config]$ mv test/dfc.properties ./
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ # With the folder empty exept for log4j.properties and dfc.properties files, dmqdocbroker is working
[dmadmin@cs-0 config]$ date; time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping; date
Wed Apr 14 07:47:17 UTC 2021
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0200.0080
Using specified port: 1489
Successful reply from docbroker at host (cs-0) on port(1490) running software version (16.4.0200.0256 Linux64).
real 0m4.280s
user 0m8.161s
sys 0m0.729s
Wed Apr 14 07:47:21 UTC 2021
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ ls -l
total 20
drwxr-x--- 8 dmadmin dmadmin 4096 Apr 14 07:47 Shared
-rw-r----- 1 dmadmin dmadmin 2153 Apr 14 07:47 dfc.keystore
-rw-rw-r-- 1 dmadmin dmadmin 481 Jul 26 2020 dfc.properties
-rwxr-x--- 1 dmadmin dmadmin 1242 Jul 26 2020 log4j.properties
drwxr-x--- 6 dmadmin dmadmin 4096 Apr 14 07:46 test
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ rm -rf Shared/ dfc.keystore
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ mv test/dfc.keystore ./
[dmadmin@cs-0 config]$ mv test/dfcfull.properties ./
[dmadmin@cs-0 config]$ mv test/dfc.properties* ./
[dmadmin@cs-0 config]$ mv test/log4j.properties* ./
[dmadmin@cs-0 config]$ mv test/ServerApps ./
[dmadmin@cs-0 config]$ mv test/Shared ./
[dmadmin@cs-0 config]$ mv test/acs ./
[dmadmin@cs-0 config]$ mv test/bpm ./
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ ls -l
total 140
drwxr-x--- 7 dmadmin dmadmin 4096 Jul 26 2020 ServerApps
drwxr-x--- 9 dmadmin dmadmin 4096 Jul 26 2020 Shared
drwxr-x--- 7 dmadmin dmadmin 4096 Jul 26 2020 acs
drwxr-x--- 7 dmadmin dmadmin 4096 Jul 26 2020 bpm
-rw-rw-r-- 1 dmadmin dmadmin 2152 Jul 26 2020 dfc.keystore
-rw-rw-r-- 1 dmadmin dmadmin 481 Jul 26 2020 dfc.properties
-rw-rw---- 1 dmadmin dmadmin 70 Jul 22 2020 dfc.properties.bak.0
-rwxr-x--- 1 dmadmin dmadmin 242 Jul 26 2020 dfc.properties.bak.1
-rw-rw-r-- 1 dmadmin dmadmin 271 Jul 26 2020 dfc.properties.bak.2
-rw-rw-r-- 1 dmadmin dmadmin 323 Jul 26 2020 dfc.properties.bak.3
-rw-rw-r-- 1 dmadmin dmadmin 481 Jul 26 2020 dfc.properties.bak.4
-rw-rw-r-- 1 dmadmin dmadmin 482 Jul 26 2020 dfc.properties.bak.5
-rwxrwx--- 1 dmadmin dmadmin 79268 Jul 22 2020 dfcfull.properties
-rwxr-x--- 1 dmadmin dmadmin 1242 Jul 26 2020 log4j.properties
drwxr-x--- 2 dmadmin dmadmin 4096 Apr 14 07:51 test
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ ls -l test/
total 0
-rwxr-x--- 1 dmadmin dmadmin 0 Jul 22 2020 dbor.properties
-rw-rw-r-- 1 dmadmin dmadmin 0 Jul 26 2020 dbor.properties.lck
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ # With the full folder except the dbor files, dmqdocbroker is still working
[dmadmin@cs-0 config]$ date; time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping; date
Wed Apr 14 07:51:30 UTC 2021
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0200.0080
Using specified port: 1489
Successful reply from docbroker at host (cs-0) on port(1490) running software version (16.4.0200.0256 Linux64).
real 0m3.501s
user 0m6.632s
sys 0m0.666s
Wed Apr 14 07:51:34 UTC 2021
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ mv test/dbor.properties* ./
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ # With the dbor files back, dmqdocbroker isn't working anymore
[dmadmin@cs-0 config]$ date; time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping; date
Wed Apr 14 07:51:56 UTC 2021
^C
real 0m30.682s
user 0m5.001s
sys 0m0.424s
Wed Apr 14 07:52:27 UTC 2021
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ mv dbor.properties.lck test/
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ # Removing just the dbor files again, dmqdocbroker is working again
[dmadmin@cs-0 config]$ date; time dmqdocbroker -t cs-0.domain.com -p 1489 -c ping; date
Wed Apr 14 07:52:36 UTC 2021
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0200.0080
Using specified port: 1489
Successful reply from docbroker at host (cs-0) on port(1490) running software version (16.4.0200.0256 Linux64).
real 0m3.185s
user 0m5.546s
sys 0m0.578s
Wed Apr 14 07:52:39 UTC 2021
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ ll
total 140
drwxr-x--- 7 dmadmin dmadmin 4096 Jul 26 2020 ServerApps
drwxr-x--- 9 dmadmin dmadmin 4096 Jul 26 2020 Shared
drwxr-x--- 7 dmadmin dmadmin 4096 Jul 26 2020 acs
drwxr-x--- 7 dmadmin dmadmin 4096 Jul 26 2020 bpm
-rwxr-x--- 1 dmadmin dmadmin 0 Jul 22 2020 dbor.properties
-rw-r----- 1 dmadmin dmadmin 0 Apr 14 07:52 dbor.properties.lck
-rw-rw-r-- 1 dmadmin dmadmin 2152 Jul 26 2020 dfc.keystore
-rw-rw-r-- 1 dmadmin dmadmin 481 Jul 26 2020 dfc.properties
-rw-rw---- 1 dmadmin dmadmin 70 Jul 22 2020 dfc.properties.bak.0
-rwxr-x--- 1 dmadmin dmadmin 242 Jul 26 2020 dfc.properties.bak.1
-rw-rw-r-- 1 dmadmin dmadmin 271 Jul 26 2020 dfc.properties.bak.2
-rw-rw-r-- 1 dmadmin dmadmin 323 Jul 26 2020 dfc.properties.bak.3
-rw-rw-r-- 1 dmadmin dmadmin 481 Jul 26 2020 dfc.properties.bak.4
-rw-rw-r-- 1 dmadmin dmadmin 482 Jul 26 2020 dfc.properties.bak.5
-rwxrwx--- 1 dmadmin dmadmin 79268 Jul 22 2020 dfcfull.properties
-rwxr-x--- 1 dmadmin dmadmin 1242 Jul 26 2020 log4j.properties
drwxr-x--- 2 dmadmin dmadmin 4096 Apr 14 07:52 test
[dmadmin@cs-0 config]$
[dmadmin@cs-0 config]$ diff dbor.properties.lck test/dbor.properties.lck
[dmadmin@cs-0 config]$
So as you can see above (it’s rather long but I wanted to put all the evidences I gathered because I still cannot believe this is the cause of the issue), just removing/renaming the empty file “dbor.properties.lck” which was there, untouched, since almost 9 months is sufficient to have the dmqdocbroker/iapi/idql working again… Trying to put back the old empty file, the issue will come back. It’s the “same” file, same content (empty), same file format, everything… The only difference would be the Inode of course and the creation/modification dates.
After some more investigations, the issue appeared to be on the NAS behind which was still having a lock on the file, somehow. For information, I also had the same behavior on a second environment but with the file “$DOCUMENTUM/config/ServerApps/identityInterprocessMutex.lock” this time… So if that even happen to you, take a look at these lock files under $DOCUMENTUM/config and make sure there are no problems with the storage.
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/MOP_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/OLS_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/GME_web-min-scaled.jpg)