Suppose that the docbroker has been stopped prematurely and that we want to shut down the repositories but the out-of-the-box dm_shutdown_repository<\/em> is not effective. Why is it so by the way ? If we look closely inside the shutdown script, we quickly notice the reason:<\/p>\n On line 29, the shutdown script first attempts to connect to the repository in order to retrieve the root pid of the server processes. On line 36, this attempt’s result is checked by the function check_connect_status defined earlier in the script at line 10. If something went wrong during the connection, iapi’s return status will be != 0 and check_connect_status will simply exit the script on line 18. So, if a repository has gone berserk, or no free sessions are available, or the docbroker is unreachable, the script will not be able to stop it. That logic is quite restrictive and we must fall back to killing the repository’s processes ourselves anyway. Ordinarily, the LIST_SESSIONS server method returns a collection containing the root_pid attribute among other valuable information, e.g.:<\/p>\n But in our case, the CS is not reachable so it cannot be queried. The number 25329 is the root_pid. It can be extracted from the log file as shown below: The extracted root_pid can be confirmed by the ps command with options ajxf showing a nice tree-like view of the running processes. E.g.:<\/p>\n On line 3, the CS for docbase dmtest was started with pid 25329 and same value for its pgid. This process started then a few child processes all with the pgid 25329. The command does not show the method server nor the docbroker as they were started separately from the CS. Putting both commands together, we get: where test is the content server to kill. This ruthless kill functionality could be added to the start-stop script listed above, either as a command-line option to the stop parameter (say, like -k as in the dm_shutdown_repository<\/em> script) or as a full parameter on a par with the stop | start | status ones, i.e.: If the content server were open source we wouldn’t have this article’s title. Instead, it would be “Forcing impromptu projections to docbrokers through signal handling in content server: an implementation” or “Shutting down a content server by sending a signal: a proposal”. We could send this request to the maintainers and probably receive a positive answer. Or we could implement the changes ourselves and submit them as a RFC. This model does not work so much in closed, commercial source which evolves following its own marketing agenda. Nonetheless, this situation gives us the opportunity to rant about it and find work-arounds. Imagine a world where all software were flawless, would it be as fun ?<\/p>\n","protected":false},"excerpt":{"rendered":" Stopping the unreachable repositories Suppose that the docbroker has been stopped prematurely and that we want to shut down the repositories but the out-of-the-box dm_shutdown_repository is not effective. Why is it so by the way ? If we look closely inside the shutdown script, we quickly notice the reason: #!\/bin\/sh ################## DOCUMENTUM SERVER SHUTDOWN FILE […]<\/p>\n","protected":false},"author":40,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[525],"tags":[],"type_dbi":[],"class_list":["post-13197","post","type-post","status-publish","format-standard","hentry","category-enterprise-content-management"],"acf":[],"yoast_head":"\n\n#!\/bin\/sh\n################## DOCUMENTUM SERVER SHUTDOWN FILE ######################\n#\n# 1994-2018 OpenText Corporation. All rights reserved\n# Version 16.4 of the Documentum Server.\n#\n# A generated server shutdown script for a repository.\n# This file was generated on Fri Aug 30 12:15:10 CEST 2019 by user dmadmin.\n#\ncheck_connect_status() {\nstatus=$1\nif [ ! $status = 0 ] ; then\n cat <<-END\n ***** $0: ERROR\n ***** Unable to complete shutdown - unable to connect\n ***** to the server to issue $2 request.\nEND\n exit 1\nfi\n}\n...\n# Stop the server\necho Stopping Documentum server for repository: [dmtestgr02]\necho ''\nDM_DMADMIN_USER=dmadmin\n#\n# Get the pid for the root process\n#\nDM_PID=`.\/iapi dmtestgr02 -U$DM_DMADMIN_USER -P -e << EOF | grep 'root_pid' | sed -e 's\/ .*[: A-Za-z]\/\/'\napply,s0,NULL,LIST_SESSIONS\nnext,s0,q0\ndump,s0,q0\nexit\nEOF`\nstatus=$?\ncheck_connect_status $status LIST_SESSIONS\n...\n kill -9 $child_pid\n...\n kill -9 $DM_PID\n...\n kill -9 $child_pid\n...\n<\/pre>\n
\nStrangely enough, the script is not scared of killing processes, it does this from several places, but it rather looks like it is a bit shy in identifying the right ones and therefore relies on the server itself or, ultimately, on the user, for help in this area.
\nAdmittedly, it is not always easy to pinpoint the right processes from the list returned by the command ps, especially if the repository is running in HA on the same machine, or if several repositories share the same machine, so extra care must be used in order not to kill the wrong ones. The dm_shutdown_docbase<\/em> avoids this difficulty altogether by asking the content server (aka CS) its root pid and that is why it aborts if it cannot contact it.
\nHistorically, the “kill” command could only “kill -9” (SIGKILL, forceful, coercive kill) but nowadays it has been generalized to send signals and could just as well have been forked to “signal” or “send”. So, can a signal be sent to the main executable ${DM_HOME}\/bin\/documentum to ask it to cleanly shut down the repository ? We wish but this has not been implemented. Signals such as SIGQUIT, SIGTRAP, SIGINT and SIGABRT are trapped indeed but will only kill the server after printing to the server’s log the last executed SQL or the call stack trace, e.g. after a SIGINT was sent:
\n
\n2019-10-11T13:14:14.045467\t24429[24429]\t0100c35080004101\tError: dm_bear_trap: Unexpected exception, (SIGINT: interrupt: (2) at (Connection Failure)), during new session creation in module dmapply.cxx after line 542. Process exiting.
\n Last SQL statement executed by DB was:
\n
\n
\n Last SQL statement executed by DB was:
\n Last SQL statement executed by DB was:
\n
\n
\n
\n
\n Last SQL statement executed by DB was:
\n
\n
\n(23962) Outer Exception handler caught exception: SIGINT: interrupt: (2) at (RPC MAIN)
\n<\/code>
\nThus, a corruption is theoretically possible while using any of those signals, just as it is when a SIGKILL signal is issued.
\nAccording to OTX Support, a trap handler that shuts down cleanly the repository has not been implemented because it needs a session to invoke the shutdown server method. OK, and what if a hidden session were opened at startup time and kept around just for such administrative cases ? How about a handler to immediately force a projection to the available docbrokers instead of waiting for the next checkpoint cycle ? As you see, there are ways to make the shutdown more resilient but my overall feeling is there is a lack of willingness to improve the content server.
\nTherefore, if waiting about 5 minutes for the repository to project to a docbroker is not acceptable, there is no other alternative than kill -9 the repository’s processes, start the docbroker(s) and then the repository. Other signals can work, but not always, and are not any safer.
\nIn order to use that command, one needs to know the content server’s root pid and since the CS does not accept any connection at this point, one must get it from another source. Once the root pid is available, it can be given to the kill command with a slight subtlety: in order to include its children processes, the root pid must be negated, e.g.:
\n
\n# use the standalone \/bin\/kill command;
\n$ \/bin\/kill --signal SIGKILL -12345
\n# or use bash's kill builtin:
\n$ command kill -s SIGKILL -12345
\n<\/code>
\nThis will transitively kill the process with pid 12345 and all the others in same group, which are the ones it started itself, directly or indirectly.
\nIf a numeric signal is preferred, the equivalent command is:
\n
\n$ \/bin\/kill -9 -12345
\n<\/code>
\nI leave it to you to decide which one is more readable.
\nSo now, we need to identify the repository’s root process. Once found, we can send its negated value the SIGKILL signal, which will propagate to all the child processes. Let’s see now how to identify this root process.<\/p>\nIdentifying the content server’s root process<\/h2>\n
\nAPI> apply,c,NULL,LIST_SESSIONS\n...\nq0\nAPI> next,c,q0\n...\nOK\nAPI> dump,c,q0\n...\nUSER ATTRIBUTES\n \n root_start : 12\/11\/2019 22:53:19\n root_pid : 25329\n shared_mem_id : 2588691\n semaphore_id : 0\n session [0]: 0100c3508000a11c\n [1]: 0100c3508000a102\n [2]: 0100c3508000a101\n db_session_id [0]: 272\n [1]: 37\n [2]: 33\n typelockdb_session_id [0]: -1\n [1]: -1\n [2]: -1\n tempdb_session_ids [0]: -1\n [1]: 45\n [2]: 36\n pid [0]: 17686\n [1]: 26512\n [2]: 26465\n user_name [0]: dmadmin\n [1]: dmadmin\n [2]: dmadmin\n user_authentication [0]: Trusted Client\n [1]: Password\n [2]: Trusted Client\n client_host [0]: docker\n [1]: 172.19.0.3\n [2]: docker\n client_lib_ver [0]: 16.4.0070.0035\n [1]: 16.4.0070.0035\n [2]: 16.4.0070.0035\n...\n<\/pre>\n
\nAn easy alternative is to simply look into the CS’s log:<\/p>\n\ndmadmin@docker:\/app\/dctm$ less \/app\/dctm\/dba\/log\/dmtest.log\n \n OpenText Documentum Content Server (version 16.4.0080.0129 Linux64.Oracle)\n Copyright (c) 2018. OpenText Corporation\n All rights reserved.\n \n2019-12-11T22:53:19.757264 25329[25329] 0000000000000000 [DM_SERVER_I_START_SERVER]info: \"Docbase dmtest attempting to open\"\n \n2019-12-11T22:53:19.757358 25329[25329] 0000000000000000 [DM_SERVER_I_START_KEY_STORAGE_MODE]info: \"Docbase dmtest is using database for cryptographic key storage\"\n...\n<\/pre>\n
\n
\n$ grep \"[DM_SERVER_I_START_SERVER]info\" \/app\/dctm\/dba\/log\/dmtest.log | gawk '{if (match($2, \/[[0-9]+]\/)) {print substr($2, RSTART + 1, RLENGTH - 2); exit}}'
\n25329
\n# or compacter:
\ngawk '{if (match($0, \/[([0-9]+)].+[DM_SERVER_I_START_SERVER]info\/, root_pid)) {print root_pid[1]; exit}}' \/app\/dctm\/dba\/log\/dmtest.log
\n25329
\n<\/code><\/p>\n\ndmadmin@docker:\/app\/dctm$ ps_gpid 25329\n PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND\n 1 25329 25329 25329 ? -1 Ss 1001 0:01 .\/documentum -docbase_name dmtest -security acl -init_file \/app\/dctm\/dba\/config\/dmtest\/server.ini\n25329 25370 25329 25329 ? -1 S 1001 0:00 _ \/app\/dctm\/product\/16.4\/bin\/mthdsvr master 0xe901fc83, 0x7f084db15000, 0x223000 50000 5 25329 dmtest \/app\/dctm\/dba\/log\n25370 25371 25329 25329 ? -1 Sl 1001 0:05 | _ \/app\/dctm\/product\/16.4\/bin\/mthdsvr worker 0xe901fc83, 0x7f084db15000, 0x223000 50000 5 0 dmtest \/app\/dctm\/dba\/log\n25370 25430 25329 25329 ? -1 Sl 1001 0:05 | _ \/app\/dctm\/product\/16.4\/bin\/mthdsvr worker 0xe901fc83, 0x7f084db15000, 0x223000 50000 5 1 dmtest \/app\/dctm\/dba\/log\n25370 25451 25329 25329 ? -1 Sl 1001 0:05 | _ \/app\/dctm\/product\/16.4\/bin\/mthdsvr worker 0xe901fc83, 0x7f084db15000, 0x223000 50000 5 2 dmtest \/app\/dctm\/dba\/log\n25370 25464 25329 25329 ? -1 Sl 1001 0:05 | _ \/app\/dctm\/product\/16.4\/bin\/mthdsvr worker 0xe901fc83, 0x7f084db15000, 0x223000 50000 5 3 dmtest \/app\/dctm\/dba\/log\n25370 25482 25329 25329 ? -1 Sl 1001 0:05 | _ \/app\/dctm\/product\/16.4\/bin\/mthdsvr worker 0xe901fc83, 0x7f084db15000, 0x223000 50000 5 4 dmtest \/app\/dctm\/dba\/log\n25329 25431 25329 25329 ? -1 S 1001 0:00 _ .\/documentum -docbase_name dmtest -security acl -init_file \/app\/dctm\/dba\/config\/dmtest\/server.ini\n25329 25432 25329 25329 ? -1 S 1001 0:00 _ .\/documentum -docbase_name dmtest -security acl -init_file \/app\/dctm\/dba\/config\/dmtest\/server.ini\n25329 25453 25329 25329 ? -1 S 1001 0:00 _ .\/documentum -docbase_name dmtest -security acl -init_file \/app\/dctm\/dba\/config\/dmtest\/server.ini\n25329 25465 25329 25329 ? -1 S 1001 0:00 _ .\/documentum -docbase_name dmtest -security acl -init_file \/app\/dctm\/dba\/config\/dmtest\/server.ini\n25329 25489 25329 25329 ? -1 S 1001 0:00 _ .\/documentum -docbase_name dmtest -security acl -init_file \/app\/dctm\/dba\/config\/dmtest\/server.ini\n25329 26439 25329 25329 ? -1 Sl 1001 0:11 _ .\/dm_agent_exec -docbase_name dmtest.dmtest -docbase_owner dmadmin -sleep_duration 0\n25329 26465 25329 25329 ? -1 S 1001 0:00 _ .\/documentum -docbase_name dmtest -security acl -init_file \/app\/dctm\/dba\/config\/dmtest\/server.ini\n 1 10112 25329 25329 ? -1 Rl 1001 0:03 .\/dm_agent_exec -docbase_name dmtest.dmtest -docbase_owner dmadmin -trace_level 0 -job_id 0800c3508000218b -log_directory \/app\/dctm\/dba\/log -docbase_id 50000\n<\/pre>\n
\nps_pgid on line 1 is a bash function defined in ~\/.bashrc as follows:<\/p>\n\n# returns the lines from ps -ajxf with given gpid;\n# the ps command's header line is printed only if at least 1 entry is found;\nfunction ps_pgid {\n pgid=$1\n ps -ajxf | gawk -v pgid=$pgid 'BEGIN {getline; header = $0; h_not_printed = 1} {if ($3 == pgid) {if (h_not_printed) {print header; h_not_printed = 0}; print}}'\n}\n<\/pre>\n
\nThus, if we execute the command below:
\n
\n$ kill --signal SIGKILL -25329
\n<\/code>
\nthe CS will be killed along with all its child processes, which is exactly what we want.<\/p>\n
\n
\nkill --signal SIGKILL -$(grep \"[DM_SERVER_I_START_SERVER]info\" \/app\/dctm\/dba\/log\/dmtest.log | gawk '{if (match($2, \/[[0-9]+]\/)) {print substr($2, RSTART + 1, RLENGTH - 2); exit}}')
\n<\/code>
\nIt may be worth defining a bash function for it too:<\/p>\n\nfunction kill_cs {\n repo=$1\n kill --signal SIGKILL -$(grep \"[DM_SERVER_I_START_SERVER]info\" \/app\/dctm\/dba\/log\/${repo}.log | gawk '{if (match($2, \/[[0-9]+]\/)) {print substr($2, RSTART + 1, RLENGTH - 2); exit}}')\n}\n \n# source it:\n. ~\/.bashrc\n \n# call it:\nkill_cs dmtest\n<\/pre>\n
\nThe naive way to search the running content server via the command “ps -ef | grep docbase_name<\/em>” can be too ambiguous in case of multiple content servers for the same repository (e.g. in a high-availability installation) or when docbase_name<\/em> is the stem of a family of docbases (e.g. dmtest_1, dmtest_2, …, dmtest_10, etc…). Besides, even if no ambiguity were possible, it would return too many processes to be killed individually. xargs could do it at once, sure, but why risk killing the wrong ones ? The above ps_pgid function is directly looking for the given group id which is the root_pid of the content server of interest taken straight out of its log file, no ambiguity here. <\/p>\nHardening start-stop.sh<\/h2>\n
\n
\nstart-stop.sh stop | start | status | kill ...
\n<\/code>
\nor, simply by deciding that a stop should always succeed and forcing a kill if needed. In such variant, the stop_docbase() function becomes:<\/p>\n\nstop_docbase() {\n echo \"stopping $docbase\"\n docbase=$1\n .\/dm_shutdown_${docbase}\n if [[ $? -eq 1 ]]; then\n echo \"killing docbase $docbase\"\n kill_cs $docbase\n fi\n echo \"docbase $docbase stopped\"\n}\n<\/pre>\nConclusion<\/h2>\n