systemd configurations for Documentum

systemd configurations for Documentum

systemd has been with us for several years now and has slowly made its way into most Linux distributions. While it has generated much controversy among sysV init hard core, the fact is that it is here to stay and we, Documentum administrators, don’t have our say in this topic. In effect, it does not impact us very much, except that a little translation work is necessary to switch to it, provided that we already went the service way. Most of the time, our custom monolithic script to stop, start and inquiry the status of the several Documentum components can be reused as-is, it is just its invocation that changes. On the other hand, we can take profit of this opportunity to refactor that big script and define separate units for each components. Since systemd lets us define dependencies between components, we can externalize these out of the script, into systemd units. As a result, our stop/start script become slenderer, more readable and easier to maintain. So let’s see how to do all this.

Invocation of the big script

Such a big, monolithic script, let’s call it documentum.sh, is executed by dmadmin and has the typical following layout:

...
start:
launch the docbrokers
start the method server
start the docbases
stop:
shut the docbases down
stop the method server
stop the docbrokers
status:
check the docbrokers
check the method server
check the docbases
...

For simplicity, let’s assume henceforth that we are logged as root when typing all the systemd-related commands below.
To invoke this script from within systemd, let’s create the documentum.service unit:

cat - <<EndOfUnit > /etc/systemd/system/documentum.service
[Unit]
Description=Documentum components controls;
Type=oneshot
RemainAfterExit=yes

ExecStart=sudo -u dmadmin -i /app/dctm/server/dbi/documentum.sh start
ExecStop=sudo -u dmadmin -i /app/dctm/server/dbi/documentum.sh stop
 
[Install]
WantedBy=multi-user.target
EndOfUnit

The clause Type is oneshot because the unit runs commands that terminate, not services.
Unlike real services whose processes keep running after they’ve been started, dm_* scripts terminate after they have done their job, which is to launch some Documentum executables as background processes; thus, RemainAfterExit is needed to tell systemd that the services are still running once started.
ExecStart and ExecStop are obviously the commands to run in order to start, respectively stop the service.
See here for a comprehensive explanation of all the unit’s directives.
Now, activate the service:

systemctl enable documentum.service

This unit has no particular dependencies because all the Documentum-related stuff is self-contained and the needed system dependencies are all available at that point.
On lines 7 and 8, root runs the big script as user dmadmin. Extra care should be taken in the “change user” command so the script is run as dmadmin. It is of paramount importance that sudo be used instead of su. Both are very closely related: a command must be executed as another user, provided the real user has the right to do so (which is the case here because systemd runs as root).

man sudo:
SUDO(8) BSD System Manager's Manual SUDO(8)
 
NAME
sudo, sudoedit — execute a command as another user
 
man su:
SU(1) User Commands SU(1)
 
NAME
su - change user ID or become superuser

However, they behave differently in relation to systemd. With “su – dmadmin -c “, the command gets attached to a session in dmadmin’s slice:

systemd-cgls
 
├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
├─user.slice
│ ├─user-618772.slice
│ │ └─session-41.scope
│ │ ├─ 5769 sshd: adm_admin2 [priv
│ │ ├─ 5783 sshd: adm_admin2@pts/
│ │ ├─ 5784 -bash
│ │ ├─11277 systemd-cgls
│ │ └─11278 less
│ └─user-509.slice
│ ├─session-c11.scope
│ │ ├─10988 ./documentum -docbase_name global_registry -security acl -init_file /app/dctm/server/dba/config/global_registry/server.ini
│ └─session-c1.scope
│ ├─6385 ./documentum -docbase_name dmtest -security acl -init_file /app/dctm/server/dba/config/dmtest/server.ini

...

Here, user id 509 is dmadmin. We see that 2 docbase processes are attached to dmadmin’s slice, itself attached to the global user.slice.
With “sudo -u dmadmin -i “, the command gets attached to the system.slice:

system-cgls
 
├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
├─user.slice
│ └─user-618772.slice
│ └─session-10.scope
│ ├─4314 sshd: adm_admin2 [priv
│ ├─4589 sshd: adm_admin2@pts/
│ ├─4590 -bash
│ ├─5927 /usr/share/centrifydc/libexec/dzdo service documentum.service start
│ ├─5928 /bin/systemctl start documentum.service
│ ├─5939 /usr/bin/systemd-tty-ask-password-agent --watch
│ ├─5940 /usr/bin/pkttyagent --notify-fd 5 --fallback
│ ├─6219 systemd-cgls
│ └─6220 less
└─system.slice
├─documentum.service
│ ├─5944 /usr/bin/sudo -u dmadmin -i /app/dctm/server/dbi/documentum.sh start
│ ├─5945 /bin/bash /app/dctm/server/dbi/documentum.sh start
│ ├─5975 ./dmdocbroker -port 1489 -init_file /app/dctm/server/dba/Docbroker.ini
│ ├─5991 ./dmdocbroker -port 1491 -init_file /app/dctm/server/dba/Docbrokerdmtest.ini
│ ├─6013 ./documentum -docbase_name global_registry -security acl -init_file /app/dctm/server/dba/config/global_registry/server.ini
│ ├─6023 ./documentum -docbase_name dmtest -security acl -init_file /app/dctm/server/dba/config/dmtest/server.ini
│ ├─6024 sleep 30
│ ├─6055 /app/dctm/server/product/7.3/bin/mthdsvr master 0xfd070016, 0x7fa02da79000, 0x223000 1000712 5 6013 global_registry /app/dctm/server/dba/log
│ ├─6056 /app/dctm/server/product/7.3/bin/mthdsvr master 0xfd070018, 0x7f261269c000, 0x223000 1000713 5 6023 dmtest /app/dctm/server/dba/log

...

Here, user 618772 ran the command “dzdo service documentum.service start” (dzdo is a Centrify command analog to sudo but with privileges checked against Active Directory) to start the documentum.service, which started the command “sudo -u dmadmin -i /app/dctm/server/dbi/documentum start” as defined in the unit and attached its processes under system.slice.
The difference is essential: at shutdown, sessions are closed abruptly, so if a stop/start script is running in it, its stop option will never be invoked.
Processes running under the system.slice on the other hand have their command’s stop option invoked properly so they can cleanly shut down.
This distinction is rarely necessary because generally all the services run as root even though their installation may be owned by some other user. E.g. an apache listening on the default port 80 must run as root. Documentum stuff was not designed to be a service, just background processes running as dmadmin. But thanks to this trick, they can still be managed as services.
At boot time, the unit will be processed and its start commands (there can be many, but here only one for the big script) executed.
It is also possible to invoke the service documentum.service manually:

systemctl start | stop | status documentum.service

The old sysvinit syntax is still available too:

service documentum.service start | stop | status

Thus, everything is in one place and uses a common management interface, which is specially appealing to a system administrator with no particular knowledge of each product installed on each machine under their control, e.g. to become dmadmin and invoke the right dm_* script.
The direct invocation of the unit file is still possible:

/etc/systemd/system/documentum.service start | stop | status

but the service interface is so much simpler.
One remark here: the status clause implemented in the big script above is not the one invoked by the command “systemctl status”:

systemctl status documentum.service
● documentum.service - Documentum Content Server controls for the runtime lifecycle
Loaded: loaded (/etc/systemd/system/documentum.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2018-10-22 14:03:09 CEST; 4min 6s ago
Process: 25388 ExecStop=/bin/su - dmadmin -c /app/dctm/server/dbi/startstop stop (code=exited, status=0/SUCCESS)
Process: 24069 ExecStart=/bin/su - dmadmin -c sh -c 'echo " ** Starting documentum"' (code=exited, status=0/SUCCESS)
Main PID: 924 (code=exited, status=0/SUCCESS)

Instead, the latter just returns the status of the service per se, not of the resources exposed by the service. It is indeed possible to display the current status of those programs in the same output but some special work need to be done for this. Basically, those processes need to periodically push their status to their service by calling systemd-notify; this could be done by a monitoring job for example. See systemd-notify’s man page for more details.
There is no ExecStatus clause in the unit either, although it would make some sense to define a command that asks the service’s processes about its status. We still need some custom script for this.

Splitting the big script

As the full systemd way is chosen, why not introduce a finer service granularity ? To do this, each Documentum component can be extracted from the big script and turned into a service of its own, as illustrated below.
Unit documentum.docbrokers.service

cat - <<EndOfUnit > /etc/systemd/system/documentum.docbrokers.service
[Unit]
Description=Documentum docbrokers controls;
Type=oneshot
RemainAfterExit=yes

# no dependencies;

# there are 2 docbrokers here;
ExecStart=sudo -u dmadmin -i /app/dctm/server/dba/dm_launch_docbroker
ExecStart=sudo -u dmadmin -i /app/dctm/server/dba/dm_launch_docbrokerdmtest
ExecStop=sudo -u dmadmin -i /app/dctm/server/dba/dm_stop_docbroker
ExecStop=sudo -u dmadmin -i /app/dctm/server/dba/dm_stop_docbrokerdmtest
 
[Install]
WantedBy=multi-user.target
EndOfUnit

Now, activate the service:

systemctl enable documentum.docbrokers.service

Lines 10 to 13 call the standard docbroker’s dm_* scripts.

Unit documentum.method-server.service

cat - <<EndOfUnit > /etc/systemd/system/documentum.method-server.service
[Unit]
Description=Documentum method server controls;

After=documentum.docbrokers.service
Requires=documentum.docbrokers.service

Type=oneshot
RemainAfterExit=yes

ExecStart=sudo -u dmadmin -i /app/dctm/server/shared/wildfly9.0.1/server/startMethodServer.sh start
ExecStop=sudo -u dmadmin -i /app/dctm/server/shared/wildfly9.0.1/server/stopMethodServer.sh stop
 
[Install]
WantedBy=multi-user.target
EndOfUnit

Now, activate the service:

systemctl enable documentum.method-server.service

While the dependency with the docbrokers is defined explicitly on line 5 and 6 (see later for an explanation of these clauses), the one with the docbases is a bit ambiguous. Traditionally, the method server is started after the docbases even though, as its name implies, it is a server for the docbases, which are thus its clients. So, logically, it should be started before the docbases, like the docbrokers, and not the other way around. However, the method server executes java code that may use the DfCs and call back into the repository, so the dependency between repositories and method server is two-way. Nevertheless, since it is the docbases that initiate the calls (methods don’t execute spontaneously on the method server), it makes sense to start the method server before the repositories and define a dependency from the latter to the former. This will also simplify the systemd configuration if a passphrase is to be manually typed to start the docbases (see paragraph below).
Lines 11 and 12 call the standard Documentum script for starting and stopping the method server.

Unit documentum.docbases.service

cat - <<EndOfUnit > /etc/systemd/system/documentum.docbases.service
[Unit]
Description=Documentum docbases controls;

After=documentum.docbrokers.service documentum.method-server.service
Requires=documentum.docbrokers.service documentum.method-server.service

Type=oneshot
RemainAfterExit=yes

ExecStart=sudo -u dmadmin -i /app/dctm/server/dba/dm_start_global_registry
ExecStart=sudo -u dmadmin -i /app/dctm/server/dba/dm_start_dmtest
ExecStop=sudo -u dmadmin -i /app/dctm/server/dbi/dm_shutdown_global_registry
ExecStart=sudo -u dmadmin -i /app/dctm/server/dba/dm_shutdown_dmtest
 
[Install]
WantedBy=multi-user.target
EndOfUnit

Now, activate the service:

systemctl enable documentum.docbases.service

Here, the dependencies must be explicitly defined because the docbases need the docbrokers to start. The method server is needed for executing java code requested by the docbases. The After= on line 5 clause says that the current unit documentum.docbases.service waits until the units listed here have been started. The Requires= clause on line 6 says that the current unit documentum.docbases.service cannot start without the other two units so they must all be started successfully, otherwise documentum.docbases.service fails. By default, they start concurrently but the After= clause postpones starting documentum.docbases.service until after the other 2 have started.
Lines 11 to 14 call the standard Documentum script for starting and stopping a docbase.
This alternative does not use the custom script any more but exclusively the ones provided by Documentum; one less thing to maintain at the cost of some loss of flexibility, should any special startup logic be required someday. Thus, don’t bin that big script so quickly, just in case.

Hybrid alternative

The custom monolithic script does everything in one place but lacks the differentiation between components. E.g. the start option starts everything and there is no way to address a single component. An enhanced script, dctm.sh, with the syntax below would be nice:

dctm.sh start|stop|status component

i.e.

dctm.sh start|stop|status docbrokers|docbases|method-server

It could even go as far as differentiating among the repositories and docbrokers:

dctm.sh start|stop|status docbroker:docbroker|docbase:docbase|method-server

A plural keyword syntax could also be used when differentiation is not wanted (or when too lazy to specify the component, or when the component’s exact name is not known/remembered), to collectively address a given type of component:

dctm.sh start|stop|status [--docbrokers|--docbroker docbroker{,docbroker}|--docbases|--docbase docbase{,docbase}|--method-server]

i.e. a list of components can be specified, or all or each of them at once. If none are specified, all of them are addressed. The highlighted target names are keywords while the italicized ones are values. This is a good exercise in parsing command-line parameters, so let’s leave it to to reader !
All these components could be addressed individually either from the corresponding service unit (or from systemd-run, see next paragraph):
Unit documentum.docbrokers.service

...
ExecStart=sudo -u dmadmin -i /app/dctm/server/dbi/dctm.sh start --docbrokers
ExecStop=sudo -u dmadmin -i /app/dctm/server/dbi/dctm.sh stop --docbrokers
...

Unit documentum.method-server.service

...
ExecStart=sudo -u dmadmin -i /app/dctm/server/dbi/dctm.sh start --method-server
ExecStop=sudo -u dmadmin -i /app/dctm/server/dbi/dctm.sh stop --method-server
...

Unit documentum.docbases.service

...
ExecStart=sudo -u dmadmin -i /app/dctm/server/dbi/dctm.sh start --docbases
ExecStop=sudo -u dmadmin -i /app/dctm/server/dbi/dctm.sh stop --docbases
...

As explained above, dctm.sh’s status parameter is not reachable from systemctl but a monitoring agent could put it to good use.
Thus, we have here the granularity of the previous alternative while retaining the flexibility of the monolithic script, e.g. for checking a status (see the next paragraph for another reason to keep the custom script). Each variant has it pros and cons and, as it is often the case, flexibility comes at the cost of complexity.

The case of the missing lockbox passphrase

If a lockbox is in use and a passphrase must be entered interactively by an administrator to start the database, then that service cannot be started by systemd at boot time because at that time the passphrase is still missing from dmadmin’s shared memory. Thus, the docbase start must be delayed until after the passphrase has been loaded. If the service’s start clause is removed and missing, systemd will complain but if we leave it, the start will effectively fail because of the missing lockbox’ passphrase. So, how to exit this dead end ?
A fake start through the clause ExecStart=/bin/true could replace the real start but then how to start the docbases via systemctl once the passphrase has been entered ?
One possible trick is to leave the invocation of the custom script in the service’s start clause but add some logic in that script so it can determine itself how its start clause was invoked. If it was within, say, a 1 minute uptime, then it is obviously an automatic invocation at boot time. The script then aborts and returns false so the service is marked “not started” and can be started manually with no need to first stop it (which would be necessary if it simply returned a 0 exit code). An administrator would then enter the lockbox passphrase, typically with the command below:

sudo -u dmadmin -i dm_crypto_boot -all –passphrase
then, type the passphrase at the prompt

and manually start the service as written above.
A possible implementation of this logic is:

start)):
MAX_BOOT_TIME=60
ut=$(read tot idle < /proc/uptime; echo ${tot%.*})
[ $ut -lt $MAX_BOOT_TIME ] && exit 1

If the service is later stopped and restarted without rebooting, the uptime would be larger than 1 minute and therefore the enhanced custom script dctm.sh (we need this one because only the docbases need to be started, the other components have been already started as services at this point) would do the start itself directly, assuming that the passphrase is now in dmadmin’s shared memory (if it’s not, the start will fail again and the service stay in the same state).
This 1 minute delay can look short but systemd attempts to start as much as possible in parallel, except when dependencies are involved, in which case some serialization is performed. This is another advantage of systemd: a shorter boot time for faster reboots. The fact that most installations run now inside virtual machines makes the reboot even faster. The delay value must not be too large because it is possible that an administrator, who may have done the shutdown themself, is waiting behind their keyboard for the reboot to complete, log in, enter the passphrase and start the service, which will be rejected by the above logic as it considers that it is too soon to do it.

Running a command as a service

systemd makes it possible to run a command as a service, in which case no unit file is necessary. This is an alternative to the missing lockbox passphrase case. An administrator would first load the passphrase in dmadmin’s shared memory and later manually invoke a custom script, with no special logic involving the uptime, as follows:

dzdo systemd-run --unit=dctm.docbases --slice=system.slice --remain-after-exit sudo -u dmadmin -i /app/dctm/server/dbi/dctm.sh start --docbases

Such services without unit files are called transient services.
Thus, only the docbrokers and the method server would have their respective unit, while the docbases would be started manually as transient services. The enhanced custom script, dctm.sh, is directly invoked here, not the documentum.docbases.service unit file (there is no need for one any more), with the special command-line argument −−docbases, as discussed in the previous paragraph.
Thanks to the parameter −−slice, the processes with be attached under system.slice and therefore be treated like a service:

├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
├─user.slice
│ └─user-618772.slice
│ └─session-10.scope
│ ├─ 4314 sshd: adm_admin2 [priv
│ ├─ 4589 sshd: adm_admin2@pts/
│ ├─ 4590 -bash
│ ├─15561 systemd-cgls
│ └─15562 systemd-cgls
└─system.slice
├─dctm.docbases.service
│ ├─15347 /usr/bin/sudo -u dmadmin -i /app/dctm/server/dbi/dctm.sh start --docbases
│ ├─15348 /bin/bash /app/dctm/server/dbi/dctm.sh start
│ ├─15378 ./dmdocbroker -port 1489 -init_file /app/dctm/server/dba/Docbroker.ini
│ ├─15395 ./dmdocbroker -port 1491 -init_file /app/dctm/server/dba/Docbrokerdmtest.ini
│ ├─15416 ./documentum -docbase_name global_registry -security acl -init_file /app/dctm/server/dba/config/global_registry/server.ini
│ ├─15426 ./documentum -docbase_name dmtest -security acl -init_file /app/dctm/server/dba/config/dmtest/server.ini

Note how “.service” has been suffixed to the given dynamic unit name dctm.docbases.
The stop and status options are available too for transient services with “systemctl stop|status dctm.docbases.service”.

Useful commands

The following systemd commands can be very useful while troubleshooting and checking the services:

systemctl --all
systemctl list-units --all
systemctl list-units --all --state=active
systemctl list-units --type=service
systemctl list-unit-files
systemctl list-dependencies documentum.docbases.service
systemctl cat documentum.docbases.service
systemctl show documentum.docbases.service
show documentum.docbases.service -p After
show documentum.docbases.service -p Before
systemctl mask ...
systemctl unmask ...
rm -r /etc/systemd/system/bad.service.d
rm /etc/systemd/system/bad.service
 
# don't forget to do this after a unit file has been edited;
systemctl daemon-reload
 
# check the journal, e.g. to verify how the processes are stopped at shutdown and restarted at reboot;
journalctl --merge
journalctl -u documentum.docbases.service
 
# reboot the machine;
/sbin/shutdown -r now

Check systemctl’s man pages for more details.

User services

All the systemd commands can be run as an ordinary user (provided the command-line option −−user is present) and services can be created under a normal account too. The unit files will be stored in the user’s ~/.config/systemd/user directory. The managing interface will be the same; it is even possible to have such user services started automatically at boot time (cf. the lingering option), and stopped at system shut down. Thus, if all we want is a smooth, no brain managing interface for the Documentum processes accessible to the unprivileged product’s administrators as dmadmin, this is a handy feature.

Conclusion

Configuring systemd-style services for Documentum is not such a big a deal once we have a clear idea of what we want.
The main advantage to go the service way is to benefit from a uniform management interface so that any administrator, even without knowledge of the product, can start it, inquiry its status, and stop it. When a passphrase to be entered interactively is in use, there is no real advantage to use a service, except to have the guarantee that its stop sequence will be invoked at shutdown so the repository will be in a consistent state at the end. Actually, for Documentum, especially in the passphrase case, going the service way or staying with a custom script or the standard dm_* scripts is more a matter of IT policies rather than a technical incentive, i.e. the final decision will be more procedural than technical. Nevertheless, having a services’ standard management interface, while still keeping custom scripts for more complicate logic, can be very convenient.