I have seen several times in the past duplicate JMS and/or ACS objects created by OpenText (and Dell/EMC before) installers. I opened several tickets with the OpenText Support over the past 4/5 years to try to get a reason and a fix (had the issue with all versions, 7.1, 7.2, 7.3, 16.4). The first few tickets (2 or 3, I don’t remember) always ended up after several months of investigation with no real solution and no real root cause found. In September 2019, I tried to open again the same kind of ticket (Service Request #4167752), related to the following issue: upon installation of a Remote CS (2nd CS), the RCS/CFS installer would always create 2 new JMS Objects in case the BPM/xCP is already installed on the Primary CS (1st CS). This time, we finally got something out of it. After a couple of months, OpenText found what they believe is the issue and they were planning on a fix in a major release because of all the ramifications this can have. So it is not really a surprise that during my last upgrade project to 16.4, the same kind of issue happened.

 

As mentioned in a previous blog about “corrupt” Lockbox, this upgrade was part of a migration from Virtual Machines to Kubernetes following the standard process of upgrade on a staging environment after a first cloning to leave the source system untouched. The source system was a CS 7.3 in High Availability with three dm_server_config objects and therefore three dm_jms_config as well as three dm_acs_config objects. However, during the repository upgrade, a fourth dm_jms_config and fourth dm_acs_config objects have been created with the exact same name as primary ones.

 

Here was the status of the objects AFTER the initial cloning to the staging environment (therefore, source hostnames are gone and replaced by the staging hostname for the Primary CS and by the target hostnames for the Remote CSs) and BEFORE the upgrade:

API> ?,c,select r_object_id, object_name, i_position*-1-1, servlet_name, supported_protocol, base_uri from dm_jms_config order by r_object_id, 3;
r_object_id       object_name                                           dm_repeating.i_position*-1-1  servlet_name                      supported_protocol  base_uri
----------------  ----------------------------------------------------  ----------------------------  --------------------------------  ------------------  --------------------------------------------------------------------------------
080f12345000105f  JMS stg_cs.domain.com:9080 for REPO1.REPO1                                       0  do_method                         https               https://stg_cs.domain.com:9082/DmMethods/servlet/DoMethod
                                                                                                   1  SAMLAuthentication                https               https://stg_cs.domain.com:9082/SAMLAuthentication/servlet/ValidateSAMLResponse
                                                                                                   2  do_mail                           https               https://stg_cs.domain.com:9082/DmMail/servlet/DoMail
080f123450001d33  JMS tgt_cs2.domain.com:9080 for REPO1.tgt_cs2_REPO1                              0  do_method                         https               https://tgt_cs2.domain.com:9082/DmMethods/servlet/DoMethod
                                                                                                   1  SAMLAuthentication                https               https://tgt_cs2.domain.com:9082/SAMLAuthentication/servlet/ValidateSAMLResponse
                                                                                                   2  do_mail                           https               https://tgt_cs2.domain.com:9082/DmMail/servlet/DoMail
080f12345006f91e  JMS tgt_cs3.domain.com:9080 for REPO1.tgt_cs3_REPO1                              0  do_method                         https               https://tgt_cs3.domain.com:9082/DmMethods/servlet/DoMethod
                                                                                                   1  SAMLAuthentication                https               https://tgt_cs3.domain.com:9082/SAMLAuthentication/servlet/ValidateSAMLResponse
                                                                                                   2  do_mail                           https               https://tgt_cs3.domain.com:9082/DmMail/servlet/DoMail
(3 rows affected)

API> ?,c,select r_object_id, object_name, acs_supported_protocol, acs_base_url from dm_acs_config order by r_object_id, acs_base_url;
r_object_id       object_name             acs_supported_protocol  acs_base_url
----------------  ----------------------  ----------------------  ------------------------------------------------
080f123450000490  stg_cs.domain.comACS1   https                   https://stg_cs.domain.com:9082/ACS/servlet/ACS
080f123450001d15  Atgt_cs2_REPO1          https                   https://tgt_cs2.domain.com:9082/ACS/servlet/ACS
080f12345006f90a  Atgt_cs3_REPO1          https                   https://tgt_cs3.domain.com:9082/ACS/servlet/ACS
(3 rows affected)

API>

 

As you can see above, all the JMS and ACS objects are using HTTPS because it’s the case in our source system. Then during the repository upgrade, the installer performs the ACS and JMS upgrade/create scripts:

[dmadmin@stg_cs ~]$ cd $DM_HOME/install/
[dmadmin@stg_cs logs]$
[dmadmin@stg_cs logs]$ grep -iE "acs|java method" install.log
09:58:28,287  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerProcessingScripts - The installer will execute the : Java Method Server update script.
09:59:33,882  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerProcessingScripts - The installer will execute the Creates ACS config object script.
10:08:36,509  INFO [main] com.documentum.fc.client.impl.connection.LiteTypeManager - Flush type dm_acs_config from maps.
10:08:36,516  INFO [main] com.documentum.fc.client.impl.connection.LiteTypeManager - Type cache refresh: dm_validation_descriptor,dmc_act_group_instance,dmc_aspect_relation,dmc_bpm_lsm,dmc_completed_workflow,dmc_completed_workitem,dmc_composite_predicate,dmc_module_config,dmc_mq_config,dmc_process_correlation_set,dmc_process_parameter,dmc_routecase_condition,dmc_transition_condition,dmc_validation_relation,dmc_wf_package_schema,dmc_wf_package_skill,dmc_wf_package_type_info,dmc_workqueue,dmc_workqueue_doc_profile,dmc_workqueue_policy,dmc_workqueue_user_profile,dmc_wpr_parent,dmc_wq_skill_info,dmc_wq_task_skill,dmc_wq_user_skill,dm_acs_config,dmc_aspect_type,dmc_java_library,dmc_jar,dmc_module,dmc_tcf_activity,dmc_tcf_activity_template,dmc_validation_module,dmc_workqueue_category,dmi_030f1234500001dc,dmc_class,dmc_constraint_set,dmc_relationship_def,dmc_config_scope_relation,dmc_preset_package,dm_ldap_config,dmi_030f1234500001e5,dmc_search_template,dmi_030f1234500001e7,dmc_calendar,dmc_calendar_event,dmc_richtext,dmc_dar,dmc_datatable,dmc_datatable_row,dmc_datatable_schema,dmc_datatable_schema_ex,dmc_datatable_settings,dmc_discussion,dmc_notepage,dmc_readcomment,dmc_comment,dmc_room,dmc_topic,
10:08:36,577  INFO [main] com.documentum.install.server.installanywhere.actions.DiWAServerRegisterAcsJmxAdmin - The Accelerated Content Services administration URL does not need to be registered.
10:08:49,481  INFO [main] com.documentum.install.server.installanywhere.actions.cfs.DiWAServerProcessingScripts2 - Executing the Java Method Server Configuration Setup script.
[dmadmin@stg_cs logs]$

 

These steps will execute the dm_acs_install.ebs, dm_jms_admin.sh, dm_jms_config_setup.ebs and other things to setup the ACS and JMS. This will create the log files dm_acs_install.out and dm_jms_config_setup.out under $DOCUMENTUM/dba/config/REPO1/. In these log files, you can see what exactly was done by these scripts during the upgrade process:

[dmadmin@stg_cs logs]$ cd $DOCUMENTUM/dba/config/REPO1
[dmadmin@stg_cs REPO1]$
[dmadmin@stg_cs REPO1]$ cat dm_jms_config_setup.out
$DM_HOME/bin/dm_jms_admin.sh -docbase REPO1.REPO1 -username dmadmin -action add,enableDFC,testDFC,migrate,dumpServerCache,listAll -jms_host_name stg_cs.domain.com -jms_port 9080 -jms_proximity 1 -webapps ServerApps -server_config_id 3d0f123450000102
2020-03-12 10:09:00 UTC:  Input arguments are: -docbase REPO1.REPO1 -username dmadmin -action add,enableDFC,testDFC,migrate,dumpServerCache,listAll -jms_host_name stg_cs.domain.com -jms_port 9080 -jms_proximity 1 -webapps ServerApps -server_config_id 3d0f123450000102
2020-03-12 10:09:00 UTC:  Input parameters are: {jms_port=[9080], server_config_id=[3d0f123450000102], docbase=[REPO1.REPO1], webapps=[ServerApps], action=[add,enableDFC,testDFC,migrate,dumpServerCache,listAll], jms_proximity=[1], jms_host_name=[stg_cs.domain.com], username=[dmadmin]}
2020-03-12 10:09:00 UTC:  ======================================================================================
2020-03-12 10:09:00 UTC:  Begin administering JMS config objects in docbase REPO1.REPO1 ...
2020-03-12 10:09:07 UTC:  The following JMS config object has been successfully created/updated in docbase REPO1
2020-03-12 10:09:07 UTC:  --------------------------------------------------------------------------------------
2020-03-12 10:09:07 UTC:                      JMS Config Name: JMS stg_cs.domain.com:9080 for REPO1.REPO1
                      JMS Config ID: 080f1234509a1ae3
                      JMS Host Name: stg_cs.domain.com
                    JMS Port Number: 9080
             Is Disabled In Docbase: F
               Repeating attributes:
               Content_Server_Id[0] = 3d0f123450000102
        Content_Server_Host_Name[0] = stg_cs.domain.com
    JMS_Proximity_Relative_to_CS[0] = 1
             Servlet to URI Mapping:
                          do_method = http://stg_cs.domain.com:9080/DmMethods/servlet/DoMethod
                 SAMLAuthentication = http://stg_cs.domain.com:9080/SAMLAuthentication/servlet/ValidateSAMLResponse
                            do_mail = http://stg_cs.domain.com:9080/DmMail/servlet/DoMail

...
2020-03-12 10:09:08 UTC:  --------------------------------------------------------------------------------------
2020-03-12 10:09:08 UTC:  Total 4 JMS Config objects found in docbase REPO1
2020-03-12 10:09:08 UTC:  --------------------------------------------------------------------------------------
...
[dmadmin@stg_cs REPO1]$
[dmadmin@stg_cs REPO1]$ cat dm_acs_install.out
Start running dm_acs_install.ebs script on docbase REPO1.REPO1
Connected to docbase REPO1.REPO1 as user dmadmin.
Create/Verify dm_acs_config type succeeded.
Create/Verify dm_bocs_config type succeeded.
Create/Verify dm_dms_config type succeeded.
Create/Verify dm_network_location_map type succeeded.
Create/Verify dm_validation_descriptor type succeeded.
Create/Verify dm_cont_transfer_config type succeeded.
Create/Verify /System/BocsConfig folder succeeded.
Create dm_acs_config object
Successfully created a dm_acs_config object for docbase REPO1.REPO1 with server name of stg_cs.domain.comACS1
Create/Verify dm_validation_descriptor data succeeded.
Create/Verify dm_cont_transfer_config object succeeded.
Create/Verify _AcsConfig_MasterChangeRecord object succeeded.
Create/Verify _NetworkLocation_MasterChangeRecord object succeeded.
Method 'dm_AsynchronousWrite' already exists
Create/Verify dm_method object dm_AsynchronousWrite succeeded.
Create/Verify dm_job object dm_AsynchronousWrite succeeded.
Method 'dm_PreCacheContent' already exists
Create/Verify dm_method object dm_PreCacheContent succeeded.
Cleanup of Cache ACS Objects Succeeded.
[dmadmin@stg_cs REPO1]$

 

These scripts actually create new objects during the upgrade with the same name as the already existing one. Checking again the repository objects AFTER the upgrade, you can see the duplicate objects with exactly the same details except for the URLs which are now in HTTP instead of HTTPS:

API> ?,c,select r_object_id, object_name, i_position*-1-1, servlet_name, supported_protocol, base_uri from dm_jms_config order by r_object_id, 3;
r_object_id       object_name                                           dm_repeating.i_position*-1-1  servlet_name                      supported_protocol  base_uri
----------------  ----------------------------------------------------  ----------------------------  --------------------------------  ------------------  --------------------------------------------------------------------------------
080f12345000105f  JMS stg_cs.domain.com:9080 for REPO1.REPO1                                       0  do_method                         https               https://stg_cs.domain.com:9082/DmMethods/servlet/DoMethod
                                                                                                   1  SAMLAuthentication                https               https://stg_cs.domain.com:9082/SAMLAuthentication/servlet/ValidateSAMLResponse
                                                                                                   2  do_mail                           https               https://stg_cs.domain.com:9082/DmMail/servlet/DoMail
080f123450001d33  JMS tgt_cs2.domain.com:9080 for REPO1.tgt_cs2_REPO1                              0  do_method                         https               https://tgt_cs2.domain.com:9082/DmMethods/servlet/DoMethod
                                                                                                   1  SAMLAuthentication                https               https://tgt_cs2.domain.com:9082/SAMLAuthentication/servlet/ValidateSAMLResponse
                                                                                                   2  do_mail                           https               https://tgt_cs2.domain.com:9082/DmMail/servlet/DoMail
080f12345006f91e  JMS tgt_cs3.domain.com:9080 for REPO1.tgt_cs3_REPO1                              0  do_method                         https               https://tgt_cs3.domain.com:9082/DmMethods/servlet/DoMethod
                                                                                                   1  SAMLAuthentication                https               https://tgt_cs3.domain.com:9082/SAMLAuthentication/servlet/ValidateSAMLResponse
                                                                                                   2  do_mail                           https               https://tgt_cs3.domain.com:9082/DmMail/servlet/DoMail
080f1234509a1ae3  JMS stg_cs.domain.com:9080 for REPO1.REPO1                                       0  do_method                         http                http://stg_cs.domain.com:9080/DmMethods/servlet/DoMethod
                                                                                                   1  SAMLAuthentication                http                http://stg_cs.domain.com:9080/SAMLAuthentication/servlet/ValidateSAMLResponse
                                                                                                   2  do_mail                           http                http://stg_cs.domain.com:9080/DmMail/servlet/DoMail
(4 rows affected)

API> ?,c,select r_object_id, object_name, acs_supported_protocol, acs_base_url from dm_acs_config order by r_object_id, acs_base_url;
r_object_id       object_name             acs_supported_protocol  acs_base_url
----------------  ----------------------  ----------------------  ------------------------------------------------
080f123450000490  stg_cs.domain.comACS1   https                   https://stg_cs.domain.com:9082/ACS/servlet/ACS
080f123450001d15  Atgt_cs2_REPO1          https                   https://tgt_cs2.domain.com:9082/ACS/servlet/ACS
080f12345006f90a  Atgt_cs3_REPO1          https                   https://tgt_cs3.domain.com:9082/ACS/servlet/ACS
080f1234509a14a1  stg_cs.domain.comACS1   http                    http://stg_cs.domain.com:9080/ACS/servlet/ACS
(4 rows affected)

API>

 

The easy way to find which objects have just been created is by checking the r_object_id or the r_creation_date obviously. Since the name of the objects will be exactly the same, you can rely on the r_object_id: the one with the higher ID will be the one created recently. Once identified, you can just destroy the duplicate objects, they aren’t needed since the original one is there already for this exact purpose.