As part of the same migration & upgrade project I talked about in previous blogs already (corrupt lockbox & duplicate objects), I have seen a strange but consistent behavior in each upgrade from Documentum 7.x to 16.x versions. During this process, just before the upgrade, a cloning is done to a staging environment, to keep the source untouched.
As part of this cloning, an update of the target_server value of the dm_job objects is done to use the new host, obviously. Therefore, once the cloning is done, I’m sure that all the jobs, without exception, were using the correct target_server, according to our best practices. This means most of the jobs targeting ANY running server and then distributed jobs with some specific configurations. As you probably already understood via the title of this blog, one of the issue I faced in every single upgrade from 7.x to 16.x is that the Documentum upgrade installer is changing the target_server of some jobs and for one of them, it is actually causing an issue.
Checking the target_server after the upgrade showed that this parameter has been updated for several jobs, which therefore lost our best practices. However, the real problem was for the dm_DataDictionaryPublisher job:
[dmadmin@stg_cs ~]$ repo="REPO1"
[dmadmin@stg_cs ~]$ iapi ${repo} -Udmadmin -Pxxx << EOC
?,c,SELECT r_object_id, object_name, r_creation_date, r_modify_date, target_server FROM dm_job WHERE object_name='dm_DataDictionaryPublisher';
exit
EOC
OpenText Documentum iapi - Interactive API interface
Copyright (c) 2018. OpenText Corporation
All rights reserved.
Client Library Release 16.4.0200.0080
Connecting to Server using docbase REPO1
[DM_SESSION_I_SESSION_START]info: "Session 010f123450373995 started for user dmadmin."
Connected to OpenText Documentum Server running Release 16.4.0200.0256 Linux64.Oracle
Session id is s0
API> r_object_id object_name r_creation_date r_modify_date target_server
---------------- -------------------------- ------------------ ------------------ ----------------------------------------------------------------------------------------
080f123450000381 dm_DataDictionaryPublisher 11/8/2018 07:04:37 2/10/2020 17:13:54 REPO1.REPO1.REPO1@stg_cs
(1 row affected)
API> Bye
[dmadmin@stg_cs ~]$
Assuming a repository name of REPO1, you can connect to it using just “${repo_name}“. However, you can also connect to a specific Content Server using “${repo_name}.${dm_server_config_name}“. For a Primary CS, it is therefore usually something like “REPO1.REPO1“. I assume that what is happening is that somewhere in the code, instead of using “REPO1” as repository name and appending “.REPO1” to it so that the job run on the Primary CS, they actually used a repository name which is already the Primary CS target and therefore you end-up with “REPO1.REPO1.REPO1@…” in the target_server, which is wrong.
During the upgrade process, there are many things happening, as you can expect… One of these is the execution of the toolset.ebs script. This script is configuring the target_server on several OOTB Documentum jobs. It will create (or rather overwrite in case of an upgrade) the log file “$DOCUMENTUM/dba/config/REPO1/sysadmin.trc“. Looking into this file about changes done to the target_server parameters, you can see the following:
[dmadmin@stg_cs ~]$ cd $DOCUMENTUM/dba/config/${repo}/
[dmadmin@stg_cs REPO1]$
[dmadmin@stg_cs REPO1]$ grep -E "dm_DataDictionaryPublisher|target_server" sysadmin.trc | sed 's,.*.[sg]et(,(,'
("set,c,080f123450000359,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f12345000035a,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f12345000035b,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f12345000035c,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f12345000035d,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f12345000035e,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f12345000035f,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f123450000360,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f123450000361,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f123450000362,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f12345000036d,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f12345000036e,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f12345000036f,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f123450000370,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f123450000371,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f123450000372,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f123450000373,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f123450000374,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f123450000375,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f123450000376,target_server","REPO1.REPO1@stg_cs") ==> true
("retrieve,c,dm_job where object_name = 'dm_DataDictionaryPublisher'") ==> "080f123450000381"
("set,c,080f123450000381,target_server","dummy") ==> true
("retrieve,c,dm_method where object_name = 'dm_DataDictionaryPublisher'") ==> "100f123450000380"
("set,c,080f123450000382,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f123450000383,target_server","REPO1.REPO1@stg_cs") ==> true
("set,c,080f123450000384,target_server","REPO1.REPO1@stg_cs") ==> true
[dmadmin@stg_cs REPO1]$
During the upgrade process, the target_server is set to “dummy” by this script for the dm_DataDictionaryPublisher job. As you know, that’s definitively not a correct value for a target_server but this is expected in this case, most probably to prevent it from running (we deactivate all the jobs before migration anyway)… Later, as part of the step (from the logs) “[main] com.documentum.install.server.installanywhere.actions.cfs.DiWAServerProcessingScripts2 – Executing the Java Method Server Configuration Setup script“, the script named “dm_defaultevent_set.ebs” is executed and create the log file “$DOCUMENTUM/dba/config/REPO1/dm_defaultevent_set.out“. Looking inside this script, you can see the following:
[dmadmin@stg_cs REPO1]$ grep -C5 "retrieve.*dm_DataDictionary" $DM_HOME/install/admin/dm_defaultevent_set.ebs
TargetServer = docbase & "." & _
dmAPIGet("get,s0,serverconfig,object_name") & _
"@" & dmAPIGet("get,s0,serverconfig,r_host_name")
ID = dmAPIGet("retrieve," & sess & ",dm_job where object_name = 'dm_DataDictionaryPublisher'")
If ID <> "" Then
buff = "set," & sess & "," & ID & ",target_server"
ret% = dmAPISet(buff, TargetServer)
buff = "set," & sess & "," & ID & ",run_now"
ret% = dmAPISet(buff, "T")
[dmadmin@stg_cs REPO1]$
Based on the above, it looks like the script is correct: it does set the target_server to be “${repo_name}.${dm_server_config_name}@${dm_server_config_hostname}“. Therefore, as I suspected & mentioned already above, it must be a problem caused by the fact that the “${repo_name}” used to execute this script isn’t “REPO1” but is instead “REPO1.REPO1“. To confirm that, simply compare the definition of the “docbase” variable from the script with the generated log file:
[dmadmin@stg_cs REPO1]$ grep "Start running" $DM_HOME/install/admin/dm_defaultevent_set.ebs
Print "Start running dm_defaultevent_set.ebs script on docbase " & docbase
[dmadmin@stg_cs REPO1]$
[dmadmin@stg_cs REPO1]$ cat dm_defaultevent_set.out
Connect to docbase REPO1.REPO1
Start running dm_defaultevent_set.ebs script on docbase REPO1.REPO1
Audited the dm_docbase_config for event dm_default_set
Setting the Data Dictionary job state to active
Successfully updated job object for DataDictionaryPublisher
Finished running dm_defaultevent_set.ebs...
Disconnect from the docbase.
[dmadmin@stg_cs REPO1]$
The above confirms that the ebs script is indeed executed with a repository name of “REPO1.REPO1” which cause the wrong target_server in the end. Looking at the other log files created by the upgrade shows that there are apparently some discrepancies. The following log files seems to be OK: dm_crypto_upgrade_52.out, headstart.out, replicate_bootstrap.out, sysadmin.trc, aso… While these ones apparently might not (would needs some debugging to validate): dm_gwm_install.out?, dm_java_methods.out?, dm_defaultevent_set.out, aso…
I guess I will open a Service Reqest with the OpenText Support to get this looked at. It might not be a big issue if it’s just about the target_server of the dm_DataDictionaryPublisher job but I’m afraid there would be something else hidden in another script or object that I haven’t seen yet… The previous blog I wrote about duplicate JMS/ACS object (link at the top of this blog) might also be a consequence? Anyway, while waiting for a feedback from OpenText, we can just fix the issue for this job by re-applying the correct configuration (running on ANY):
[dmadmin@stg_cs REPO1]$ iapi ${repo} -Udmadmin -Pxxx << EOC
?,c,UPDATE dm_job OBJECTS SET target_server=' ' WHERE object_name='dm_DataDictionaryPublisher';
?,c,SELECT r_object_id, object_name, r_creation_date, r_modify_date, target_server FROM dm_job WHERE object_name='dm_DataDictionaryPublisher';
exit
EOC
OpenText Documentum iapi - Interactive API interface
Copyright (c) 2018. OpenText Corporation
All rights reserved.
Client Library Release 16.4.0200.0080
Connecting to Server using docbase REPO1
[DM_SESSION_I_SESSION_START]info: "Session 010f123450373a42 started for user dmadmin."
Connected to OpenText Documentum Server running Release 16.4.0200.0256 Linux64.Oracle
Session id is s0
API> objects_updated
---------------
1
(1 row affected)
[DM_QUERY_I_NUM_UPDATE]info: "1 objects were affected by your UPDATE statement."
API> ?,c,SELECT r_object_id, object_name, r_creation_date, r_modify_date, target_server FROM dm_job WHERE object_name='dm_DataDictionaryPublisher';
r_object_id object_name r_creation_date r_modify_date target_server
---------------- -------------------------- ------------------ ------------------ --------------
080f123450000381 dm_DataDictionaryPublisher 11/8/2018 07:04:37 2/11/2020 07:04:45
(1 row affected)
API> Bye
[dmadmin@stg_cs REPO1]$
If I receive some important information from the OpenText Service Request, I will try to update this blog to add the details (like other potential issues linked).
Edit: I opened the SR#4485076 and OpenText confirmed the bug. They apparently already knew about the issue for dm_DataDictionaryPublisher since it has been fixed (normally) in 16.4 P23 released on 30-March-2020. They are still looking into potential issues with the other scripts.
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/MOP_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/OLS_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/GME_web-min-scaled.jpg)