I often see customers with Virtual Machines or Containers that have IPv6 disabled. I assume this is mainly for simplicity, since most services are usually only running internally, and organizations don’t really need more IPs than what IPv4 can provide to begin with. In this blog, I will talk about an upgrade project from Documentum 20.2 to 23.4, using containers on Kubernetes running on the same base OS image (rhel 8.x). With the exact same configuration (OS and Documentum), the 20.2 Connection Broker works correctly, but the 23.4 one refuses to start.

As part of the upgrade process, a new image with Documentum 23.4 binaries has been created from the same base OS image. Then, the environment has been started on Kubernetes with the new image, to initiate the upgrade process from 20.2 to 23.4. The first thing done is the upgrade of the Connection Broker, which, according to the logs, went successfully. However, when the process tried to start, it quickly disappeared as it failed with the following:

[dmadmin@cs-0 ~]$ cat $DOCUMENTUM/dba/log/docbroker.*.1489.log
OpenText Documentum Connection Broker (version 23.4.0000.0143  Linux64)
Copyright (c) 2023. OpenText Corporation
2024-10-08T10:57:59.236419 [DM_DOCBROKER_I_START]info:  "Docbroker has started.  Process id: 8428"
2024-10-08T10:57:59.245348 [DM_DOCBROKER_E_START_NETWISE]error:  "The docbroker failed to start due to an error returned by the Netwise networking subsystem.  Failed API: nl_open().  Error specifics: (601) System error(97) in call to socket().   errno: 97, message: Address family not supported by protocol."
[dmadmin@cs-0 ~]$

The message “Address family not supported by protocol” is usually present when something tries to start on IPv6 while it is disabled. This is the kind of message you might get on a Java application (like the Java Method Server) if you don’t specify the “-Djava.net.preferIPv4Stack=true” JVM parameter.

So, I went ahead and checked the logs and configuration file of the previous Connection Broker that was in 20.2:

[dmadmin@cs-0 ~]$ ### 20.2
[dmadmin@cs-0 ~]$ cat $DOCUMENTUM/dba/log/docbroker.*.1489.log
OpenText Documentum Connection Broker (version 20.2.00013.0135  Linux64)
Copyright (c) 2020. OpenText Corporation
2024-09-07T06:40:37.224228 [DM_DOCBROKER_I_START]info:  "Docbroker has started.  Process id: 6867"
2024-09-07T06:40:37.225776 [DM_DOCBROKER_I_REGISTERED_PORT]info:  "The Docbroker registered using port (1489)."
2024-09-07T06:40:37.226591 [DM_DOCBROKER_I_LISTENING]info:  "The Docbroker is listening on network address: (INET_ADDR: family: 2, port: 1489, host: cs-0 (172.20.20.139, 8b1414ac))"
2024-09-07T06:40:37.227683 [DM_DOCBROKER_I_REGISTERED_PORT]info:  "The Docbroker registered using port (1490)."
2024-09-07T06:40:37.228659 [DM_DOCBROKER_I_LISTENING]info:  "The Docbroker is listening on network address: (INET_ADDR: family: 2, port: 1490, host: cs-0 (172.20.20.139, 8b1414ac))"
[dmadmin@cs-0 ~]$
[dmadmin@cs-0 ~]$ cat $DOCUMENTUM/dba/Docbroker.ini
[DOCBROKER_CONFIGURATION]
secure_connect_mode=dual
[dmadmin@cs-0 ~]$
[dmadmin@cs-0 ~]$ cat $DOCUMENTUM/dba/dm_documentum_config.txt
[DOCBROKER_Docbroker]
NAME=Docbroker
VERSION=20.2.0000.0110
PORT=1489
[DOCBASE_gr_dbi]
NAME=gr_dbi
VERSION=20.2.0000.0110
DATABASE_CONN=dctmdb
[dmadmin@cs-0 ~]$

And just to be sure, I compared the configuration file with its updated version after upgrading to 23.4:

[dmadmin@cs-0 ~]$ ### 23.4
[dmadmin@cs-0 ~]$ cat $DOCUMENTUM/dba/Docbroker.ini
[DOCBROKER_CONFIGURATION]
secure_connect_mode=dual
[dmadmin@cs-0 ~]$
[dmadmin@cs-0 ~]$ cat $DOCUMENTUM/dba/dm_documentum_config.txt
[DOCBROKER_Docbroker]
NAME=Docbroker
VERSION=23.4.0000.0143
PORT=1489
[DOCBASE_gr_dbi]
NAME=gr_dbi
VERSION=20.2.0000.0110
DATABASE_CONN=dctmdb
[dmadmin@cs-0 ~]$

Obviously, there are no differences in the “Docbroker.ini” file and only the version has been updated by the installer in the “dm_documentum_config.txt” file, as expected. What is surprising is that a Documentum 20.2 Connection Broker can detect, by itself, that IPv6 is disabled and starts with IPv4 (without any errors), while a 23.4 Connection Broker can’t do that.

A quick look at the OS configuration definitely confirms that IPv6 is disabled at the OS level, as “sysctl” does not know of any parameter related to “ipv6“: there is nothing returned by the command except the “permission denied” which can be ignored (it’s because I don’t have root access inside the container). Same thing for the IPv6 module parameter for “disable“, it is set to true:

[dmadmin@cs-0 ~]$ sysctl -a | grep ipv6
sysctl: permission denied on key 'kernel.apparmor_display_secid_mode'
sysctl: permission denied on key 'kernel.apparmor_restrict_unprivileged_io_uring'
sysctl: permission denied on key 'kernel.apparmor_restrict_unprivileged_userns_complain'
sysctl: permission denied on key 'kernel.apparmor_restrict_unprivileged_userns_force'
sysctl: permission denied on key 'kernel.cad_pid'
sysctl: permission denied on key 'kernel.unprivileged_userns_apparmor_policy'
sysctl: permission denied on key 'kernel.usermodehelper.bset'
sysctl: permission denied on key 'kernel.usermodehelper.inheritable'
sysctl: permission denied on key 'net.ipv4.tcp_fastopen_key'
sysctl: permission denied on key 'vm.mmap_rnd_bits'
sysctl: permission denied on key 'vm.mmap_rnd_compat_bits'
sysctl: permission denied on key 'vm.stat_refresh'
[dmadmin@cs-0 ~]$
[dmadmin@cs-0 ~]$ cat /sys/module/ipv6/parameters/disable
1
[dmadmin@cs-0 ~]$

What should be done then? As for the JMS, the easy way out is to force Documentum to start on the IPv4 address only. For that purpose, there is a specific parameter that can be added into the Connection Broker and the Repository configuration files.

To force the Connection Broker to use IPv4 only, you can define the “host” parameter in the “[DOCBROKER_CONFIGURATION]” section of the “Docbroker.ini” file. If you set the value to the current IPv4 address, then the Connection Broker will be able to start again:

[dmadmin@cs-0 ~]$ cat $DOCUMENTUM/dba/Docbroker.ini
[DOCBROKER_CONFIGURATION]
secure_connect_mode=dual
[dmadmin@cs-0 ~]$
[dmadmin@cs-0 ~]$ ip="172.20.20.125"
[dmadmin@cs-0 ~]$ sed -i "/^\[DOCBROKER_CONF.*/a host=${ip}" $DOCUMENTUM/dba/Docbroker.ini
[dmadmin@cs-0 ~]$
[dmadmin@cs-0 ~]$ cat $DOCUMENTUM/dba/Docbroker.ini
[DOCBROKER_CONFIGURATION]
host=172.20.20.125
secure_connect_mode=dual
[dmadmin@cs-0 ~]$
[dmadmin@cs-0 ~]$ $DOCUMENTUM/dba/dm_launch_Docbroker
starting connection broker on current host: [cs-0]
with connection broker log: [$DOCUMENTUM/dba/log/docbroker.cs-0.1489.log]
connection broker pid: 7415
[dmadmin@cs-0 ~]$
[dmadmin@cs-0 ~]$ cat $DOCUMENTUM/dba/log/docbroker.*.1489.log
OpenText Documentum Connection Broker (version 23.4.0000.0143  Linux64)
Copyright (c) 2023. OpenText Corporation
2024-10-08T11:03:41.924702 [DM_DOCBROKER_I_START]info:  "Docbroker has started.  Process id: 7415"

[RPCMAI] Got the IPV4 address dmNetwork::isValidIPV4(172.20.20.125)
2024-10-08T11:03:41.925179 [DM_DOCBROKER_I_REGISTERED_PORT]info:  "The Docbroker registered using port (1489)."
2024-10-08T11:03:41.925237 [DM_DOCBROKER_I_LISTENING]info:  "The Docbroker is listening on network address: (INET_ADDR: family: 2, port: 1489, host: 172.20.20.125 (172.20.20.125, 7d1414ac))"
2024-10-08T11:03:41.925271 [DM_DOCBROKER_I_REGISTERED_PORT]info:  "The Docbroker registered using port (1490)."
2024-10-08T11:03:41.925311 [DM_DOCBROKER_I_LISTENING]info:  "The Docbroker is listening on network address: (INET_ADDR: family: 2, port: 1490, host: 172.20.20.125 (172.20.20.125, 7d1414ac))"
[dmadmin@cs-0 ~]$

As you can see from the log, we are forcing the “host” to a specific IP, whereas before it was “cs-0“. This means that every time the IP changes, you must also update the Connection Broker configuration file, which can be annoying if you do not automate this process.

As a side note, if you already have a “host” parameter in the “[TRANSLATION]” section (c.f. for Hostname Translation), then it looks like the above won’t be needed. Even if it’s a different section and a different parameter (“host=internal_ip” vs “host=external_ip=internal_ip“), as long as you have at least one “host” parameter in the configuration file, it’s enough for the Connection Broker to understand that it needs to use IPv4 only, and therefore to start properly.

As far as I could see, the problem was only with the Connection Broker. The Repository was able to start without any additional configuration. However, if you want to align the components, then you can do so by defining the “ip_mode” parameter in the “[SERVER_STARTUP]” section of the “server.ini” file. For example, it can be placed just below the “install_owner“:

[dmadmin@cs-0 ~]$ grep -A1 "^install_owner" $DOCUMENTUM/dba/config/gr_dbi/server.ini
install_owner = dmadmin

[dmadmin@cs-0 ~]$
[dmadmin@cs-0 ~]$ sed -i "/^install_owner.*/a ip_mode = V4ONLY" $DOCUMENTUM/dba/config/${docbase}/server.ini
[dmadmin@cs-0 ~]$
[dmadmin@cs-0 ~]$ grep -A1 "^install_owner" $DOCUMENTUM/dba/config/gr_dbi/server.ini
install_owner = dmadmin
ip_mode = V4ONLY
[dmadmin@cs-0 ~]$

In summary, I suspect that OpenText has made a small change to the way the Connection Broker starts, most likely around address validation. But this seems to break the startup if IPv6 is disabled while there is no “host” defined in the configuration file. The fix is simple, but I guess it’s still something I could call a regression…

Post Views: 289
Thumbnail [60x60]
by
Morgan Patou