A few weeks ago at a customer, I was installing a completely new sandbox environment for one of our Application Teams. On this environment, we used a new version stack with the most recent patch set (at that time): CS 7.2 P16, D2 4.6 P05, aso… A new patch level means a lot of “fun” trying to install and execute the updated installers of Documentum/BPM/D2… And of course a lot of new bugs ;). In this blog I will talk about a new issue we never faced before that is related to the Lockbox libraries.

 

With the older version stack, when we installed D2, we always kept the D2 Lockbox libraries under $DOCUMENTUM/d2-lib/lockbox on the CS and then we referenced these libraries in the environment variable ($PATH, $LD_LIBRARY_PATH and $CLASSPATH) of our Installation Owner. In the documentation, it is not asked to set these variables as “permanent” by adding them in the bash_profile for example or something similar but this is what we used to do to avoid adding them again and again each time we needed to update the D2 Lockbox.

 

For this new version stack and since the Official Documentation didn’t change, we kept the same approach. After installing D2, we had to restart the Content Server for a maintenance at the OS level. As you probably already know if you are familiar with the CS 7.2, there is also a Lockbox specific to the CS that is normally stored under $DOCUMENTUM/dba/secure/. Whenever you restart the host, you will need to store the lockbox inside the Shared Memory and that’s how it is done:

[dmadmin@content_server_01 ~]$ pwlockbox="L0ckb0xP4ssphr4se"
[dmadmin@content_server_01 ~]$ pwaek="A3kP4ssphr4se"
[dmadmin@content_server_01 ~]$ 
[dmadmin@content_server_01 ~]$ dm_crypto_boot -lockbox lockbox.lb -lockboxpassphrase $pwlockbox -passphrase $pwaek -all

Please wait. This will take a few seconds ...

Please wait, this will take a few seconds..
Setting up the (single) passphrase for all keys in the shared memory region..
Operation succeeded
[dmadmin@content_server_01 ~]$ 
[dmadmin@content_server_01 ~]$ 
[dmadmin@content_server_01 ~]$ dm_crypto_manage_lockbox -lockbox lockbox.lb -lockboxpassphrase $pwlockbox -resetfingerprint
Lockbox lockbox.lb
Lockbox Path $DOCUMENTUM/dba/secure/lockbox.lb
Reset host done

 

These two commands are working properly so the Lockbox has been put in the Shared Memory – normally – and the docbases should now be able to start. As explained before, we are referencing the D2 Lockbox libraries inside the environment variable by default and not the CS specific versions. So I tried to start the docbase as always and checked the logs to ensure that it was running properly:

[dmadmin@content_server_01 ~]$ $DOCUMENTUM/dba/dm_start_GR_DOCBASE
starting Documentum server for repository: [GR_DOCBASE]
with server log: [$DOCUMENTUM/dba/log/GR_DOCBASE.log]
server pid: 7056
[dmadmin@content_server_01 ~]$
[dmadmin@content_server_01 ~]$ cat $DOCUMENTUM/dba/log/GR_DOCBASE.log
The Lockbox file is newer than the active Lockbox library. Retry with a newer version of the Lockbox library.
The Lockbox file is newer than the active Lockbox library. Retry with a newer version of the Lockbox library.
2016-09-29T12:02:38.314624      7056[7056]      0000000000000000        [DM_CRYPTO_F_KEYSTORE_INIT]fatal:  "Failed to initialize keystore at $DOCUMENTUM/dba/secure/CSaek. Internal error - 1057226514"

[dmadmin@content_server_01 ~]$
[dmadmin@content_server_01 ~]$

 

So that’s the error I wanted to share in this blog. As you can see, the docbase wasn’t able to start because the lockbox file is “newer” than the active Lockbox libraries. So I started to do some tests to try to understand what was happening exactly and how to solve it.

 

Ok so the first test I did is removing the D2 Lockbox libraries from the environment variables (.bash_profile + reload of shell session) and then try to start the docbase again:

[dmadmin@content_server_01 ~]$
[dmadmin@content_server_01 ~]$ exit
logout
[morganpatou@content_server_01 ~]$ su - dmadmin
[dmadmin@content_server_01 ~]$
[dmadmin@content_server_01 ~]$
[dmadmin@content_server_01 ~]$ pwlockbox="L0ckb0xP4ssphr4se"
[dmadmin@content_server_01 ~]$ pwaek="A3kP4ssphr4se"
[dmadmin@content_server_01 ~]$ dm_crypto_boot -lockbox lockbox.lb -lockboxpassphrase $pwlockbox -passphrase $pwaek -all

Please wait. This will take a few seconds ...

Please wait, this will take a few seconds..
Setting up the (single) passphrase for all keys in the shared memory region..
Operation succeeded
[dmadmin@content_server_01 ~]$ 
[dmadmin@content_server_01 ~]$ 
[dmadmin@content_server_01 ~]$ dm_crypto_manage_lockbox -lockbox lockbox.lb -lockboxpassphrase $pwlockbox -resetfingerprint
Lockbox lockbox.lb
Lockbox Path $DOCUMENTUM/dba/secure/lockbox.lb
Lockbox open failed The Lockbox file is newer than the active Lockbox library. Retry with a newer version of the Lockbox library.
** Operation failed **
[dmadmin@content_server_01 ~]$

 

As you can see, after removing the D2 Lockbox libraries from the environment, the lockbox can’t be stored in the Shared Memory anymore… So based on this information, it is clear that the Content Server Lockbox libraries (stored under $DM_HOME/bin) are different than the D2 Lockbox libraries (in our case stored under $DOCUMENTUM/d2-lib/lockbox)… It’s “fun” that two patches that have been released exactly at the same time are using different versions of the Lockbox libraries… The annoying point here is that the CS Lockbox isn’t readable anymore using the CS Lockbox libraries and that can be checked using the following command:

[dmadmin@content_server_01 ~]$ dm_crypto_create -lockbox lockbox.lb -lockboxpassphrase $pwlockbox -keyname CSaek -passphrase $pwaek -algorithm AES_256_CBC -check

The Lockbox file is newer than the active Lockbox library. Retry with a newer version of the Lockbox library.
** No AEK store exists in lockbox lockbox.lb and got status code returned as '1057226514'.
[dmadmin@content_server_01 ~]$ 
[dmadmin@content_server_01 ~]$

 

The problem I can see here is that when installing D2, the CS Lockbox has apparently somehow been corrupted by the D2 Lockbox libraries. The fact is that when a docbase is trying to start, the start script (dm_start_GR_DOCBASE) is setting the environment using the script $DM_HOME/bin/dm_set_server_env.sh which overwrites our environment variables because this file is using the CS Lockbox libraries. That’s the reason why the docbase wasn’t able to start earlier: the docbase is trying to start using the CS libraries (older) while the lockbox has been tempered with the D2 libraries (newer).

 

There are several ways to quickly workaround this specific issue and I will try to describe them below.

 

First solution: Create a copy of the dm_crypto_create and dm_crypto_create.bin files to force it to use the environment variable defined in the .bash_profile and not reload them based on the dm_set_server_env.sh (removing the 3 lines related to the LD_LIBRARY_PATH):

[dmadmin@content_server_01 ~]$ cd $DM_HOME/bin
[dmadmin@content_server_01 bin]$ cp dm_crypto_create my_dm_crypto_create
[dmadmin@content_server_01 bin]$ cp dm_crypto_create.bin my_dm_crypto_create.bin
[dmadmin@content_server_01 bin]$ vi my_dm_crypto_create
[dmadmin@content_server_01 bin]$ diff dm_crypto_create my_dm_crypto_create
21,23d20
< else
<   LD_LIBRARY_PATH=${location}
<   export LD_LIBRARY_PATH

 

Once this is done, you can compare the result of the check command:

[dmadmin@content_server_01 bin]$ ./dm_crypto_create -lockbox lockbox.lb -lockboxpassphrase $pwlockbox -keyname CSaek -passphrase $pwaek -algorithm AES_256_CBC -check

The Lockbox file is newer than the active Lockbox library. Retry with a newer version of the Lockbox library.
** No AEK store exists in lockbox lockbox.lb and got status code returned as '1057226514'.
[dmadmin@content_server_01 bin]$
[dmadmin@content_server_01 bin]$
[dmadmin@content_server_01 bin]$ ./my_dm_crypto_create -lockbox lockbox.lb -lockboxpassphrase $pwlockbox -keyname CSaek -passphrase $pwaek -algorithm AES_256_CBC -check

Key - CSaek uses algorithm AES_256_CBC.

** An AEK store with the given passphrase exists in lockbox lockbox.lb and got status code returned as '0'.
[dmadmin@content_server_01 bin]$
[dmadmin@content_server_01 bin]$

 

=> As you can see, this is now working with the updated script but that’s not really a sexy solution…

 

 

Second solution: Update the file dm_set_server_env.sh to force it to use the D2 Lockbox libraries instead of the CS ones. You can check the Official Documentation about how to setup environment variables for D2 and then just put that at the end of this file. In our case, this was something like:

[dmadmin@content_server_01 bin]$ tail -6 $DM_HOME/bin/dm_set_server_env.sh
export LD_LIBRARY_PATH=$DOCUMENTUM/d2-lib/lockbox/lib/native/linux_gcc34_x64:$LD_LIBRARY_PATH
export PATH=$DOCUMENTUM/d2-lib/lockbox/lib/native/linux_gcc34_x64:$PATH
export CLASSPATH=$DOCUMENTUM/d2-lib/D2.jar:$DOCUMENTUM/d2-lib/LB.jar:$DOCUMENTUM/d2-lib/LBJNI.jar:$CLASSPATH
# The environment that exists after this script executes is output to
# the dm_set_server_env.log file in $DOCUMENTUM_SHARED/logs
env >> $DOCUMENTUM_SHARED/logs/dm_set_server_env.log

 

This solution is a little bit better but as soon as you will upgrade/migrate your Content Server, this file might be overwritten by the Patch and therefore you will face this issue again… 🙁

 

The solution N°2 is what EMC recommended us to do so that’s what we did in the first place and we were able to start the docbase, aso… But one week later, we tried to install a new docbase and then the Server Configuration Program wasn’t able to read the lockbox anymore and therefore we weren’t able to create any new docbase! This is an extract from the file Server Configuration Program log file:

13:58:10,266  INFO [AWT-EventQueue-0] com.documentum.install.server.ui.panels.DiWPServerUseExistingLockBox - UserSelection: "lockbox.lb"
13:58:10,266  INFO [AWT-EventQueue-0] com.documentum.install.server.ui.panels.DiWPServerUseExistingLockBox - UserSelection: "AES_256_CBC"
13:58:10,266  INFO [AWT-EventQueue-0] com.documentum.install.server.ui.panels.DiWPServerUseExistingLockBox - UserInput: "CSaek"
13:58:10,266  INFO [AWT-EventQueue-0] com.documentum.install.server.ui.panels.DiWPServerUseExistingLockBox - UserInputPassword: "**********"
13:58:10,266  INFO [AWT-EventQueue-0] com.documentum.install.server.ui.panels.DiWPServerUseExistingLockBox - UserSelection: "true"
13:58:10,266  INFO [AWT-EventQueue-0] com.documentum.install.server.ui.panels.DiWPServerUseExistingLockBox - UserInput: "lockbox.lb"
13:58:10,266  INFO [AWT-EventQueue-0] com.documentum.install.server.ui.panels.DiWPServerUseExistingLockBox - UserInputPassword: "**********"
13:58:10,507 ERROR [Thread-97] com.documentum.install.server.installanywhere.actions.DiWAServerValidateLockboxPassphrase - Check AEK key passphrase failed

 

Therefore the solutions 1 and 2 are actually not good because we are able to start the docbase but not to install a new one… So we had to find another, better solution and it all started with this question: which component(s) is/are actually using the D2 Lockbox on the Content Server? Of course the answer to this question is: only the Java Method Server.

 

Based on this information, we therefore decided to restrict the access to the D2 Lockbox libraries only to the JMS. This has been done by removing the environment variables mentioned above from the bash_profile and/or from the dm_set_server_env.sh and then by updating the JMS startup script with the same:

[dmadmin@content_server_01 ~]$ head -6 $DOCUMENTUM_SHARED/jboss7.1.1/server/startMethodServer.sh
#!/bin/sh

#for D2.lockbox
export LD_LIBRARY_PATH=$DOCUMENTUM/d2-lib/lockbox/lib/native/linux_gcc34_x64:$LD_LIBRARY_PATH
export PATH=$DOCUMENTUM/d2-lib/lockbox/lib/native/linux_gcc34_x64:$PATH
export CLASSPATH=$DOCUMENTUM/d2-lib/D2.jar:$DOCUMENTUM/d2-lib/LB.jar:$DOCUMENTUM/d2-lib/LBJNI.jar:$CLASSPATH

 

After that, we “simply” regenerated the Content Server Lockbox with the CS libraries (basically you need to remove completely the current encryption inside the docbase and force the CS to recreate it from scratch… Arnaud Berbier will most probably write a blog on this subject in the next few days/weeks). Then we restarted the whole Content Server and everything was working properly with the right Lockbox libraries and we were able to create new docbases.