In a Documentum upgrade project from 20.x to 23.4 earlier this year, I faced an issue which prevented the installation/upgrade of D2. The problem was located inside the D2-DAR while it was trying to update the workflow trackers, which failed because of an ORA-01427 error. The exact error displayed was this one:
[emc.installer] dmbasic output : [DM_SERVER_E_EXEC_SQL]error: "The execsql method failed due to a database error: ORA-01427: single-row subquery returns more than one row"
Before that happened, I successfully upgraded 7 or 8 environments from 20.x to 23.4, for that customer, and after that, I did around 20 more. However, this issue only happened once, so not sure what exactly was wrong, and the Application/Business Teams couldn’t find what really caused the initial issue (inside the DB) either… In any cases, to start my investigation, I tried to quickly compare the generated logs for the D2-DAR installation on a working vs non-working Repository.
Working Repository logs:
###
## Working repo:
###
[dmadmin@cs-0 DARsInternal]$ cat dar-deploy-D2-DAR.dar-REPO1-20250703-133024.log
[INFO] ******************************************************
[INFO] * Headless Composer
[INFO] * Version: 23.4.0000.0120
[INFO] * Java version: 17.0.13 (64bit)
[INFO] * Java home: $JAVA_HOME
[INFO] * Set storage type: false
[INFO] *
[INFO] * DAR file: $DM_HOME/install/DARsInternal/D2-DAR.dar
[INFO] * Project name: D2-DAR
[INFO] * Built by Composer: 23.4.0000.0027
[INFO] *
...
[INFO] dmbasic output : Finished creating required indexes on D2 types.
[INFO] dmbasic output : Copied workflow_id in to d2c_workflow_tracker object
[INFO] dmbasic output : Copied process_id in to d2c_workflow_tracker object
[INFO] dmbasic output : Copied supervisor_name in to d2c_workflow_tracker object
[INFO] dmbasic output : Adding d2_digital_sign_tracker_user user to the group d2_digital_signature_tracker
[INFO] dmbasic output : Fetched the group d2_digital_signature_tracker
[INFO] dmbasic output : Check for 'Public searches' folder to migrate.
[INFO] dmbasic output : Folder '/Resources/D2/[MIGRATE] Public searches' not found. Do not migrate.
[INFO] dmbasic output : Disconnected from the server.
[INFO] Finished executing post-install script Thu Jul 03 13:32:13 UTC 2025
[INFO] Project 'D2-DAR' was successfully installed.
[dmadmin@cs-0 DARsInternal]$
Non-working Repository logs:
###
## Non working repo:
###
[dmadmin@cs-0 DARsInternal]$ cat dar-deploy-D2-DAR.dar-REPO2-20250704-144219.log
[INFO] ******************************************************
[INFO] * Headless Composer
[INFO] * Version: 23.4.0000.0120
[INFO] * Java version: 17.0.13 (64bit)
[INFO] * Java home: $JAVA_HOME
[INFO] * Set storage type: false
[INFO] *
[INFO] * DAR file: $DM_HOME/install/DARsInternal/D2-DAR.dar
[INFO] * Project name: D2-DAR
[INFO] * Built by Composer: 23.4.0000.0027
[INFO] *
...
[INFO] dmbasic output : Finished creating required indexes on D2 types.
[INFO] dmbasic output : Copied workflow_id in to d2c_workflow_tracker object
[INFO] dmbasic output : [DM_SERVER_E_EXEC_SQL]error: "The execsql method failed due to a database error: ORA-01427: single-row subquery returns more than one row"
[INFO] dmbasic output :
[INFO] dmbasic output : [DM_TYPE_MGR_E_INDEX_ALREADY_EXISTS]error: "The index being created on type d2_recently_used_profile already exists. The existing index is represented by dmi_index object with id 1f012345800005a0."
[INFO] dmbasic output :
[INFO] dmbasic output : [DM_TYPE_MGR_E_INDEX_ALREADY_EXISTS]error: "The index being created on type d2_recently_used_profile already exists. The existing index is represented by dmi_index object with id 1f0123458000059f."
[INFO] dmbasic output :
[INFO] dmbasic output :
[INFO] dmbasic output : Exiting with dmExit(-1)
[ERROR] Procedure execution failed with dmbasic exit value : 255
[INFO] D2-DAR install failed.
[ERROR] Unable to install dar file $DM_HOME/install/DARsInternal/D2-DAR.dar
com.emc.ide.installer.PostInstallException: Error running post-install procedure "PostInstall". Please contact the procedure owner to verify if it is functioning properly.
at internal.com.emc.ide.installer.DarInstaller.postInstall(DarInstaller.java:1574)
at internal.com.emc.ide.installer.DarInstaller.doInstall(DarInstaller.java:669)
...
Caused by: com.emc.ide.external.dfc.procedurerunner.ProcedureRunnerException: Procedure execution failed with dmbasic exit value : 255
at com.emc.ide.external.dfc.procedurerunner.ProcedureRunnerUtils.executeDmBasic(ProcedureRunnerUtils.java:285)
at com.emc.ide.external.dfc.procedurerunner.ProcedureRunner.execute(ProcedureRunner.java:55)
...
[ERROR] Failed to install DAR
Unable to install dar file $DM_HOME/install/DARsInternal/D2-DAR.dar
at com.emc.ant.installer.api.InstallerAntTask.installDar(InstallerAntTask.java:273)
at com.emc.ant.installer.api.InstallerAntTask.execute(InstallerAntTask.java:135)
...
Caused by: com.emc.ide.installer.PostInstallException: Error running post-install procedure "PostInstall". Please contact the procedure owner to verify if it is functioning properly.
at internal.com.emc.ide.installer.DarInstaller.postInstall(DarInstaller.java:1574)
at internal.com.emc.ide.installer.DarInstaller.doInstall(DarInstaller.java:669)
...
Caused by: com.emc.ide.external.dfc.procedurerunner.ProcedureRunnerException: Procedure execution failed with dmbasic exit value : 255
at com.emc.ide.external.dfc.procedurerunner.ProcedureRunnerUtils.executeDmBasic(ProcedureRunnerUtils.java:285)
at com.emc.ide.external.dfc.procedurerunner.ProcedureRunner.execute(ProcedureRunner.java:55)
at internal.com.emc.ide.installer.DarInstaller.postInstall(DarInstaller.java:1570)
... 42 more
As often with DAR installation failures, there aren’t a lot of information. It’s clear that the installation failed because there is apparently some wrong data inside the DB (a query that should return only 1 row is giving more results), but there isn’t much more details expect that. By comparing the 2 different logs, you can understand that the issue is located between these 2 lines:
[INFO] dmbasic output : Copied workflow_id in to d2c_workflow_tracker object
[INFO] dmbasic output : Copied process_id in to d2c_workflow_tracker object
Therefore, finding the root cause is essentially finding what is being done between these 2 lines. A DAR is essentially a zip (Documentum Archive), so you can just extract it and look at its content to see what is being done. In this case (D2 23.4), I could find the necessary source code inside the file named “D2-DAR/bin/content/55/-1569930955/runnableContent.crtext“. Here is an extract of that file:
Print "Copied workflow_id in to d2c_workflow_tracker object"
sQuery = "update d2c_workflow_tracker_s set process_id = " & _
"(select child_id from dm_relation_s where dm_relation_s.parent_id = " & _
"d2c_workflow_tracker_s.r_object_id and dm_relation_s.relation_name = " & _
"'D2_WF_TRACKER_TEMPLATE') where (process_id IS NULL or process_id = '0000000000000000') " & _
" and exists (select child_id from dm_relation_s where dm_relation_s.parent_id = " & _
"d2c_workflow_tracker_s.r_object_id and dm_relation_s.relation_name = 'D2_WF_TRACKER_TEMPLATE')"
bRet = dmAPIExec("execsql," & sess & "," & sQuery)
if (not bRet) then
Call Err_Handle(sess, "0")
Print "Exiting with dmExit(-1)"
dmExit(-1)
end if
Print "Copied process_id in to d2c_workflow_tracker object"
Therefore, the query that is causing the issue is the one referenced above as “sQuery”, i.e. this one, transformed into plain SQL:
update d2c_workflow_tracker_s
set process_id=(select child_id from dm_relation_s
where dm_relation_s.parent_id=d2c_workflow_tracker_s.r_object_id
and dm_relation_s.relation_name='D2_WF_TRACKER_TEMPLATE')
where (process_id IS NULL or process_id='0000000000000000')
and exists (select child_id from dm_relation_s
where dm_relation_s.parent_id=d2c_workflow_tracker_s.r_object_id
and dm_relation_s.relation_name='D2_WF_TRACKER_TEMPLATE');
And indeed, executing this query produces the same result:
SQL> update d2c_workflow_tracker_s
set process_id=(select child_id from dm_relation_s
where dm_relation_s.parent_id=d2c_workflow_tracker_s.r_object_id
and dm_relation_s.relation_name='D2_WF_TRACKER_TEMPLATE')
where (process_id IS NULL or process_id='0000000000000000')
and exists (select child_id from dm_relation_s
where dm_relation_s.parent_id=d2c_workflow_tracker_s.r_object_id
and dm_relation_s.relation_name='D2_WF_TRACKER_TEMPLATE');
update d2c_workflow_tracker_s set process_id=(select child_id from dm_relation_s where dm_relation_s.parent_id=d2c_workflow_tracker_s.r_object_id and dm_relation_s.relation_name='D2_WF_TRACKER_TEMPLATE') where (process_id IS NULL or process_id='0000000000000000') and exists (select child_id from dm_relation_s where dm_relation_s.parent_id=d2c_workflow_tracker_s.r_object_id and dm_relation_s.relation_name='D2_WF_TRACKER_TEMPLATE')
*
ERROR at line 1:
ORA-01427: single-row subquery returns more than one row
Unfortunately, because of the way the query is formed (mapping all d2c_workflow_tracker_s.process_id 1-by-1 from the dm_relation_s.child_id), you cannot just execute the sub-query (whatever is present between the parenthesis):
SQL> select child_id from dm_relation_s
where dm_relation_s.parent_id=d2c_workflow_tracker_s.r_object_id
and dm_relation_s.relation_name='D2_WF_TRACKER_TEMPLATE';
select child_id from dm_relation_s where dm_relation_s.parent_id=d2c_workflow_tracker_s.r_object_id and dm_relation_s.relation_name='D2_WF_TRACKER_TEMPLATE'
*
ERROR at line 1:
ORA-00904: "D2C_WORKFLOW_TRACKER_S"."R_OBJECT_ID": invalid identifier
Even if you “correct” it (by adding the 2nd table name in the “from”), then it will just display something that isn’t directly usable:
SQL> select child_id from dm_relation_s, d2c_workflow_tracker_s
where dm_relation_s.parent_id=d2c_workflow_tracker_s.r_object_id
and dm_relation_s.relation_name='D2_WF_TRACKER_TEMPLATE';
CHILD_ID
----------------
4b01234580044d0f
4b01234580057910
4b01234580002572
...
4b01234580057910
4b01234580057910
4b01234580311d91
1406 rows selected.
What to do then? Well, you need to go further. There are 1406 rows from the above query (number of dm_process being tracked or to be more accurate, the number of relations between a Workflow Tracker (parent_id) and its dm_process (child_id)). Assuming that the error is legit, then that would probably mean that there could be less Workflow Trackers than that number. Therefore, after a bit of reflection, I executed the following query to try to find all Workflow Trackers and the count of relations that each and every one of them has. Under normal conditions, each Workflow Tracker should have a single relation (meaning a single dm_process (1-by-1 mapping)) but that wasn’t the case:
SQL> select d2c_workflow_tracker_s.r_object_id, count(dm_relation_s.parent_id)
from d2c_workflow_tracker_s, dm_relation_s
where dm_relation_s.parent_id=d2c_workflow_tracker_s.r_object_id
and dm_relation_s.relation_name='D2_WF_TRACKER_TEMPLATE'
group by d2c_workflow_tracker_s.r_object_id
order by 2;
R_OBJECT_ID COUNT(DM_RELATION_S.PARENT_ID)
---------------- ------------------------------
09012345800df7eb 1
09012345800df7ed 1
...
0901234580313e31 1
090123458004fdde 2
090123458005cdf5 2
090123458005fb44 2
1403 rows selected.
There are only 1403 Workflow Trackers, which means that there are 3 “additional” relations (or 3 dm_process which are assigned to an already-used Workflow Tracker). To get more details on the Workflow and their Workflow Trackers, these kinds of queries can be executed:
SQL> select parent_id, child_id from dm_relation_s
where parent_id IN ('090123458004fdde','090123458005cdf5','090123458005fb44')
and relation_name='D2_WF_TRACKER_TEMPLATE'
order by 1, 2;
PARENT_ID CHILD_ID
---------------- ----------------
090123458004fdde 4b01234580002575
090123458004fdde 4b01234580057910
090123458005cdf5 4b01234580057910
090123458005cdf5 4b01234580057910
090123458005fb44 4b01234580057910
090123458005fb44 4b01234580057910
SQL> select r_object_id, object_name, r_creator_name from dm_sysobject_sp
where r_object_id IN ('090123458004fdde','090123458005cdf5','090123458005fb44');
R_OBJECT_ID OBJECT_NAME R_CREATOR_NAME
---------------- ---------------------------------------------------------------------- -----------------
090123458004fdde Start Import Approval October 29, 2021 at 10:34:45 AM UTC OQSOP01
090123458005cdf5 Start Ready For Approval November 11, 2021 at 3:55:04 PM UTC OQAuthor01
090123458005fb44 Start In Review November 16, 2021 at 12:01:32 PM UTC OQSOP01
SQL> select r_object_id, object_name from dm_sysobject_sp
where r_object_id IN ('4b01234580002575','4b01234580057910');
R_OBJECT_ID OBJECT_NAME
---------------- ----------------------
4b01234580002575 WF_Import
4b01234580057910 WF_Review_Approve
SQL> select * from dm_relation_sp
where parent_id='090123458005cdf5'
and relation_name='D2_WF_TRACKER_TEMPLATE';
R_OBJECT_ID RELATION_NAME PARENT_ID CHILD_ID CHILD_LABEL PERMANENT_LINK ORDER_NO EFFECTIVE EXPIRATIO DESCRIPTION I_PARTITION I_IS_REPLICA I_VSTAMP
---------------- -------------------------------- ---------------- ---------------- -------------------------------- -------------- ---------- --------- --------- --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ----------- ------------ ----------
370123458000d92a D2_WF_TRACKER_TEMPLATE 090123458005cdf5 4b01234580057910 0 0 01-JAN-01 01-JAN-01 0 0 0
370123458000cd4f D2_WF_TRACKER_TEMPLATE 090123458005cdf5 4b01234580057910 0 0 01-JAN-01 01-JAN-01 0 0 0
Clearly, there is a Data/Business problem. Two of the three Workflow Trackers have an exact duplicate relation for the same dm_process while the third one tracks two different dm_process, which is even more surprising.
To be able to proceed, I noted all the IDs and informed the Application/Business teams and then I did what the SQL command from the D2-DAR was supposed to do. Therefore, I took one of the 2 values and assigned it to the process_id of the Workflow Tracker:
SQL> update d2c_workflow_tracker_s
set process_id='4b01234580002575'
where (process_id IS NULL or process_id='0000000000000000')
and r_object_id='090123458004fdde';
1 row updated.
SQL> update d2c_workflow_tracker_s
set process_id='4b01234580057910'
where (process_id IS NULL or process_id='0000000000000000')
and r_object_id='090123458005fb44';
1 row updated.
SQL> update d2c_workflow_tracker_s
set process_id='4b01234580057910'
where (process_id IS NULL or process_id='0000000000000000')
and r_object_id='090123458005cdf5';
1 row updated.
After that, I re-triggered the query which was previously failing and this time it worked properly for all the remaining Workflow Trackers:
SQL> update d2c_workflow_tracker_s
set process_id=(select child_id from dm_relation_s
where dm_relation_s.parent_id=d2c_workflow_tracker_s.r_object_id
and dm_relation_s.relation_name='D2_WF_TRACKER_TEMPLATE')
where (process_id IS NULL or process_id='0000000000000000')
and exists (select child_id from dm_relation_s
where dm_relation_s.parent_id=d2c_workflow_tracker_s.r_object_id
and dm_relation_s.relation_name='D2_WF_TRACKER_TEMPLATE');
1400 rows updated.
I was then able to start the upgrade pipeline from scratch again, and it was able to complete properly. As previously stated, the Application/Business teams still didn’t find what caused these 3 rows, but it’s only a DEV, so I can only assume that it was some human errors at some points, like a wrong DQL being executed that duplicated some values or something similar. In any cases, what is interesting for this blog is the investigation process and the way to get to the bottom of things by searching inside the D2 DAR and analyzing the DB content.