At a customer having a lot of Documentum environments, I recently faced an interesting issue while upgrading an RCS/CFS to 20.2. The same could potentially reproduce during a standard installation of a RCS/CFS, under certain circumstances (in case there are still some things left in the repository after a removal of the said RCS/CFS and then reinstallation for example, you will see why below). While trying to run the RCS/CFS configurator, it failed with the below error:

[dmadmin@cs-1 ~]$ cat $DM_HOME/install/logs/install.log
14:02:25,184  INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - The product name is: CfsConfigurator
14:02:25,184  INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - The product version is: 20.2.0000.0110
14:02:25,184  INFO [main]  -
14:02:25,217  INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - Done InitializeSharedLibrary ...
14:02:25,237  INFO [main] com.documentum.install.multinode.cfs.installanywhere.actions.DiWAServerCfsInitializeImportantServerVariables - The installer is gathering system configuration information.
...
14:04:02,083  INFO [main] com.documentum.install.server.installanywhere.actions.cfs.DiWAServerProcessingScripts2 - Executing the Docbase HeadStart script.
14:04:10,378 ERROR [main] com.documentum.install.server.installanywhere.actions.cfs.DiWAServerProcessingScripts2 - The installer failed to execute the Docbase HeadStart script. For more information, please read output file: $DOCUMENTUM/dba/config/MyRepoName001/headstart.out.
com.documentum.install.shared.common.error.DiException: The installer failed to execute the Docbase HeadStart script. For more information, please read output file: $DOCUMENTUM/dba/config/MyRepoName001/headstart.out.
        at com.documentum.install.server.installanywhere.actions.cfs.DiWAServerProcessingScripts2.setup(DiWAServerProcessingScripts2.java:98)
        at com.documentum.install.shared.installanywhere.actions.InstallWizardAction.install(InstallWizardAction.java:73)
        at com.zerog.ia.installer.actions.CustomAction.installSelf(Unknown Source)
        at com.zerog.ia.installer.AAMgrBase.an(Unknown Source)
        at com.zerog.ia.installer.ConsoleBasedAAMgr.ac(Unknown Source)
        at com.zerog.ia.installer.AAMgrBase.am(Unknown Source)
        ...
        at com.zerog.lax.LAX.launch(Unknown Source)
        at com.zerog.lax.LAX.main(Unknown Source)
14:04:10,381  INFO [main]  - The INSTALLER_UI value is SILENT
14:04:10,381  INFO [main]  - The KEEP_TEMP_FILE value is true
...
[dmadmin@cs-1 ~]$
[dmadmin@cs-1 ~]$ cat $DOCUMENTUM/dba/config/MyRepoName001/headstart.out
docbaseNameOnly = MyRepoName001
Connected to docbase MyRepoName001.cs-1_MyRepoName001 as user dmadmin.
Running headstart.ebs script on docbase MyRepoName001.cs-1_MyRepoName001 to create a remote content server.
The link for *_cs16.4* file has been created successfully
Default Storage Area exists
Create Default File Storage Area
[DM_STORAGE_E_LOCK_FILE_LOCATION_NOT_UNIQUE]error:  "the file store fs_rcs_cs-1_MyRepoName001 cannot be created because file store (280f450a80000114) is using location /data/dctm/distributed/cs-1_MyRepoName001/MyRepoName001/content_storage_01"

Failed to create Default File Storage Area
[dmadmin@cs-1 ~]$

As you might know, each RCS/CFS will have its own distributed dm_filestore named “fs_rcs_${dm_server_config_name}” that will use a dm_location named “storage_01_${dm_server_config_name}“, by default. Here, the error appears to be that this distributed dm_filestore cannot be created because another dm_filestore (280f450a80000114) is already using the distributed dm_location… Therefore, I checked the Repository objects on the Primary CS because I found that quite strange:

[dmadmin@cs-0 ~]$ iapi $DOCBASE_NAME -Udmadmin -Pxxx << EOC
> ?,c,select r_object_id, name, root from dm_filestore where r_object_id='280f450a80000114';
> ?,c,select r_object_id, object_name, file_system_path from dm_location where object_name IN (select root from dm_filestore where r_object_id='280f450a80000114');
> EOC

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2020. OpenText Corporation
        All rights reserved.
        Client Library Release 20.2.0000.0082

Connecting to Server using docbase MyRepoName001
[DM_SESSION_I_SESSION_START]info:  "Session 010f450a808431fb started for user dmadmin."

Connected to OpenText Documentum Server running Release 20.2.00013.0135  Linux64.Oracle
Session id is s0
API> r_object_id       name                             root
----------------  -------------------------------  ----------------------------------------------------------------
280f450a80000114  trusted_content_store_01         trusted_content_storage_01
(1 row affected)

API> r_object_id       object_name                      file_system_path
----------------  -------------------------------  ------------------------------------------------------------
3a0f450a822691dc  trusted_content_storage_01       /data/dctm/MyRepoName001/trusted_content_storage_01
(1 row affected)

API> Bye
[dmadmin@cs-0 ~]$

As you can see above, the object mentioned in the error message doesn’t seem to have anything to do with the distributed dm_filestore/dm_location. Therefore, it’s probably Documentum that tried to put a “useful” message but failed to do so miserably. In any cases, I then tried to check and see if the distributed dm_filestore/dm_location are already there or not:

[dmadmin@cs-0 ~]$ iapi $DOCBASE_NAME -Udmadmin -Pxxx << EOC
> ?,c,select r_object_id, name, root from dm_filestore where name like 'fs_rcs_%';
> ?,c,select r_object_id, name, root from dm_filestore where root like 'storage_01_%';
> ?,c,select r_object_id, object_name, file_system_path from dm_location where object_name like 'storage_01_%';
> EOC

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2020. OpenText Corporation
        All rights reserved.
        Client Library Release 20.2.0000.0082

Connecting to Server using docbase MyRepoName001
[DM_SESSION_I_SESSION_START]info:  "Session 010f450a808431fc started for user dmadmin."

Connected to OpenText Documentum Server running Release 20.2.00013.0135  Linux64.Oracle
Session id is s0
API> r_object_id       name                             root
----------------  -------------------------------  ----------------------------------------------------------------
(0 rows affected)

API> r_object_id       name                             root
----------------  -------------------------------  ----------------------------------------------------------------
(0 rows affected)

API> r_object_id       object_name                      file_system_path
----------------  -------------------------------  ------------------------------------------------------------
3a0f450a80001d0a  storage_01_cs-1_MyRepoName001    /data/dctm/distributed/cs-1_MyRepoName001/MyRepoName001/content_storage_01
3a0f450a8031150a  storage_01_cs-2_MyRepoName001    /data/dctm/distributed/cs-2_MyRepoName001/MyRepoName001/content_storage_01
(2 rows affected)

API> Bye
[dmadmin@cs-0 ~]$

What the above result means is that there is no distributed dm_filestore and there is no dm_location related to it (obviously, since there is nothing to related to) but there is already a correct distributed dm_location, it’s just that it’s not used by anything…

Therefore, it appears that someone or something deleted the distributed dm_filestore (or it was never created in the first place) and this is the reason for this issue. The only solution, from my point of view, was to recreate the missing dm_filestore that is needed for the RCS. It’s possible to do in iAPI/iDQL but since I had a Documentum Administrator (DA) available, I just used it:

  1. Login to the required Repository through DA using dmadmin
  2. Navigate to Administration > Storage
  3. Click on File > New > File Store
  4. Set the Name to: fs_rcs_cs-1_MyRepoName001 (replace “cs-1_MyRepoName001” with the name of your dm_server_config for this RCS/CFS)
  5. Set the Location to: storage_01_cs-1_MyRepoName001 (the one that already exists for this RCS/CFS)
  6. Leave all other options as default
  7. Click on OK

Unfortunately, DA also displayed an error message and it’s the same error as the one from the headstart:

Failed to create or save the dm_filestore 
[DM_STORAGE_E_LOCK_FILE_LOCATION_NOT_UNIQUE]error:  "the file store fs_rcs_cs-1_MyRepoName001 cannot be created because file store (280f450a80000114) is using location /data/dctm/distributed/cs-1_MyRepoName001/MyRepoName001/content_storage_01"

The next thing I tried to do is to rename the distributed folder on the File System to see if the creation of the dm_filestore would then work:

[dmadmin@cs-0 ~]$ cd /data/dctm/distributed/
[dmadmin@cs-0 distributed]$
[dmadmin@cs-0 distributed]$ ls -l cs-1_MyRepoName001/MyRepoName001/content_storage_01/000f450a/dm_location_owner.lck
-rw------- 1 dmadmin dmadmin 16 Apr 16  2020 cs-1_MyRepoName001/MyRepoName001/content_storage_01/000f450a/dm_location_owner.lck
[dmadmin@cs-0 distributed]$
[dmadmin@cs-0 distributed]$ mv cs-1_MyRepoName001 cs-1_MyRepoName001.backup
[dmadmin@cs-0 distributed]$

Trying again to create the dm_filestore by simply clicking on OK again:

Failed to create or save the dm_filestore 
[DM_STORAGE_E_MKDIR_FAILED]error:  "Cannot create storage directory /data/dctm/distributed/cs-1_MyRepoName001/MyRepoName001/content_storage_01/000f450a"

Therefore, I put back the folder in its place and instead, just removed the final folder instead:

[dmadmin@cs-0 distributed]$ mv cs-1_MyRepoName001.backup cs-1_MyRepoName001
[dmadmin@cs-0 distributed]$
[dmadmin@cs-0 distributed]$ mv cs-1_MyRepoName001/MyRepoName001/content_storage_01/000f450a/ ./
[dmadmin@cs-0 distributed]$

And finally, the dm_filestore could be created properly:

I created the second one as well since there are three CS on this environment (so two RCS/CFS) and then the Repository objects appeared to be correct afterwards:

[dmadmin@cs-0 distributed]$ iapi $DOCBASE_NAME -Udmadmin -Pxxx << EOC
> ?,c,select r_object_id, name, root from dm_filestore where name like 'fs_rcs_%';
> ?,c,select r_object_id, object_name, file_system_path from dm_location where object_name IN (select root from dm_filestore where name like 'fs_rcs_%');
> EOC

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2020. OpenText Corporation
        All rights reserved.
        Client Library Release 20.2.0000.0082

Connecting to Server using docbase MyRepoName001
[DM_SESSION_I_SESSION_START]info:  "Session 010f450a80846588 started for user dmadmin."

Connected to OpenText Documentum Server running Release 20.2.00013.0135  Linux64.Oracle
Session id is s0
API> r_object_id       name                             root
----------------  -------------------------------  ----------------------------------------------------------------
280f450a8000011f  fs_rcs_cs-1_MyRepoName001        storage_01_cs-1_MyRepoName001
280f450a80000120  fs_rcs_cs-2_MyRepoName001        storage_01_cs-2_MyRepoName001
(2 rows affected)

API> r_object_id       object_name                      file_system_path
----------------  -------------------------------  ------------------------------------------------------------
3a0f450a8031150a  storage_01_cs-2_MyRepoName001    /data/dctm/distributed/cs-2_MyRepoName001/MyRepoName001/content_storage_01
3a0f450a80001d0a  storage_01_cs-1_MyRepoName001    /data/dctm/distributed/cs-1_MyRepoName001/MyRepoName001/content_storage_01
(2 rows affected)

API> Bye
[dmadmin@cs-0 distributed]$

As often, the Documentum error messages are not very explicit… In any cases, going back to the RCS and after manually re-creating the distributed dm_filestore, the RCS Upgrade was successful without any error this time:

[dmadmin@cs-1 ~]$ cat $DM_HOME/install/logs/install.log
22:12:06,464  INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - The product name is: CfsConfigurator
22:12:06,465  INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - The product version is: 20.2.0000.0110
22:12:06,465  INFO [main]  -
22:12:06,548  INFO [main] com.documentum.install.shared.installanywhere.actions.InitializeSharedLibrary - Done InitializeSharedLibrary ...
22:12:06,582  INFO [main] com.documentum.install.multinode.cfs.installanywhere.actions.DiWAServerCfsInitializeImportantServerVariables - The installer is gathering system configuration information.
...
22:14:23,375  INFO [main] com.documentum.install.server.installanywhere.actions.cfs.DiWAServerProcessingScripts2 - Executing the Docbase HeadStart script.
22:15:01,973  INFO [main] com.documentum.install.server.installanywhere.actions.cfs.DiWAServerProcessingScripts2 - Executing the Creates ACS config object script.
22:15:17,250  INFO [main] com.documentum.install.server.installanywhere.actions.cfs.DiWAServerProcessingScripts2 - Executing the This script does miscellaneous setup tasks for remote documentum servers script.
22:15:52,303  INFO [main] com.documentum.install.server.installanywhere.actions.cfs.DiWAServerProcessingScripts2 - Executing the Creates the necessary objects for enabling email template feature. script.
22:16:27,374  INFO [main] com.documentum.install.server.installanywhere.actions.cfs.DiWAServerProcessingScripts2 - Executing the Creates default JMS config object script.
22:17:50,969  INFO [main] com.documentum.install.multinode.cfs.installanywhere.actions.DiWAServerCfsRegisterAcsJmxAdmin - The Accelerated Content Services administration URL does not need to be registered.
22:17:50,971  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAStartAppServerInstance - starting appserver instance.
22:17:50,971  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - Running $JMS_HOME/server/startMethodServer.sh
22:18:21,028  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - The batch file: $DOCUMENTUM/temp/installer/wildfly/dctm_tmpcmd0.sh exist? false
22:18:21,028  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - The user home is : /home/dmadmin
22:18:21,032  INFO [main] com.documentum.install.appserver.jboss.JbossApplicationServer - Executing temporary batch file: $DOCUMENTUM/temp/installer/wildfly/dctm_tmpcmd0.sh for running: ./jboss-cli.sh -c --commands="read-attribute server-state" --controller=localhost:9084
22:18:32,844  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAStartAppServerInstance - starting appserver instance done.
[dmadmin@cs-1 ~]$
[dmadmin@cs-1 ~]$ cat $DOCUMENTUM/dba/config/MyRepoName001/headstart.out
docbaseNameOnly = MyRepoName001
Connect to docbase MyRepoName001.MyRepoName001 as user dmadmin.
Start running headstart.ebs script on docbase MyRepoName001.MyRepoName001
Update install owner's failed_auth_attempt attribute
Default Storage Area exists
Default File Storage Area exists
Default Default Share Area exists
Default Common Area exists
Default Event Area exists
Default Log Area exists
...
Finished running HEADSTART.ebs...
Disconnect from the docbase.
[dmadmin@cs-1 ~]$

Therefore, the RCS/CFS was properly upgraded. While looking deeper into the issue, I looked at the dozens of Documentum environments that this customer has, and I found that ~5% of the environments have the exact same issue or a similar one. The “similar” issue is that from time to time, some of the distributed dm_filestore are present BUT with a different name than what it is supposed to have… I found a few environments without any distributed dm_filestore (exact same issue as this blog) and I also found some environment with a dm_filestore like “fs_rc_${dm_server_config_name}” (missing ‘s’), some others with “fs_r_${dm_server_config_name}” (missing ‘cs’) and finally some others with “fs__${dm_server_config_name}” (missing ‘rcs’)…

As you can see, the normal name got cut and after looking at it, it appeared that it was depending on the length of the “${dm_server_config_name}” value/name… For the environments with a problem, the full dm_filestore name never exceeded 21 characters. Therefore, if the dm_server_config_name was longer than 14 characters, the “fs_rcs_” in the name would be reduced to “fs_rc_” if 15 chars, to “fs_r_” if 16 chars, to “fs__” if 17 chars and the dm_filestore would completely disappear (not be present at all) if the dm_server_config_name was equal or longer than 18 chars… All the environments with the issue were installed initially in Documentum 7.x and then upgraded later to 16.4 or 20.2, no environments installed directly in 16.4 or 20.2 had a similar issue, even if their dm_server_config_name is longer than 18 chars, it would still be present with a correct name… At the moment, that’s all I could find so there is a good probability of this issue being linked to previous Documentum 7.x versions or to a previous upgrade that would somehow corrupt/rename the dm_filestore object. That’s something I will have to keep an eye on in the future!

Post Views: 1,914
Thumbnail [60x60]
by
Morgan Patou