If you planning to upgrade your Oracle RAC Clusterware from version 19.30 or higher to release 23.26.1 please be aware that this will not work as there is issue with de-supported qosmserver and OC4J resource come in 19.30 update. To avoid problem described below, install 23.26.4 which bypass this issue during upgrade.
Always upgrade to version which was released the same time or later then your Older release patched version.
For example as 23.26.1 was releases earlier then 19.30 of Clusterware it will case you troubles during patching, so always get newest version for now its October 2026, Release Update 23.26.4
Below I will describe how to rollback in case you face issue with your clusterware upgrade.
during upgrade from 19.31 >23.26.1 rootupgrade.sh script will fail in step 6 with error CLSRSC-1009: failed to stop resource OC4J
[root@rac001_lab:~]# /u01/app/23.26.1/grid/rootupgrade.sh
Check /u01/app/23.26.1/grid/install/root_rac001_lab_2026-11-03_15-43-44-143253259.log for the output of root script
[root@rac001_lab:~]# cat /u01/app/23.26.1/grid/install/root_rac001_lab_2026-11-03_15-43-44-143253259.log
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/23.26.1/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
RAC option enabled on: Linux
Executing command '/u01/app/23.26.1/grid/perl/bin/perl -I/u01/app/23.26.1/grid/perl/lib -I/u01/app/23.26.1/grid/crs/install /u01/app/23.26.1/grid/crs/install/rootcrs.pl -upgrade'
Using configuration parameter file: /u01/app/23.26.1/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/oracle/crsdata/rac001_lab/crsconfig/crsupgrade_rac001_lab_2026-11-03_03-43-44PM.log
2026/11/03 15:43:50 CLSRSC-595: Executing upgrade step 1 of 16: 'UpgradeTFA'.
2026/11/03 15:43:50 CLSRSC-4015: Performing install or upgrade action for Oracle Autonomous Health Framework (AHF).
2026/11/03 15:43:50 CLSRSC-4012: Shutting down Oracle Autonomous Health Framework (AHF).
2026/11/03 15:44:07 CLSRSC-4013: Successfully shut down Oracle Autonomous Health Framework (AHF).
2026/11/03 15:44:07 CLSRSC-595: Executing upgrade step 2 of 16: 'ValidateEnv'.
2026/11/03 15:44:07 CLSRSC-595: Executing upgrade step 3 of 16: 'GetOldConfig'.
2026/11/03 15:44:07 CLSRSC-692: Checking whether CRS entities are ready for upgrade. This operation may take a few minutes.
2026/11/03 15:44:08 CLSRSC-693: CRS entities validation completed successfully.
2026/11/03 15:44:08 CLSRSC-464: Starting retrieval of the cluster configuration data
CRS-4672: Successfully backed up the Voting File for Cluster Synchronization Service.
2026/11/03 15:44:11 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2026/11/03 15:44:11 CLSRSC-595: Executing upgrade step 4 of 16: 'UpgPrechecks'.
2026/11/03 15:44:11 CLSRSC-595: Executing upgrade step 5 of 16: 'SetupOSD'.
2026/11/03 15:44:11 CLSRSC-595: Executing upgrade step 6 of 16: 'PreUpgrade'.
2026/11/03 15:44:12 CLSRSC-1009: failed to stop resource OC4J
Died at /u01/app/23.26.1/grid/crs/install/oragqosaserver.pm line 304.
So if rootupgrade failed and exit in this stage, you need to first downgrade your CRS to ‘NORMAL’ state as it would be [ROLLING UPGRADE] mode , which prevent you from future operations, check your clusterware active version
Questions about your own setup? A dbi services consultant can look at your case.
Talk to an expert (opens in a new tab)
oracle@rac001_lab:/u01/app/19.31.0.0/grid/ [+ASM1] crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [ROLLING UPGRADE]. The cluster active patch level is [3307946645].For RAC Clusterware there are diffrent scenarios for downgrage and rollback depends which stage issue happend.
In this case its not so hard, however need downtime for whole cluster. downgrade with –online option did not work in my case
step 1. Start CRS using the 19c Grid Infrastructure home if down on node
/u01/app/19.31.0.0/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.step 2. Stop ASM rolling migration on both ASM instances nodes:
oracle@rac001_lab:~/sqlplus "/as sysasm"
SQL*Plus: Release 19.0.0.0.0 - Production on Wed Nov 4 11:05:16 2026
Version 19.31.0.0.0
Copyright (c) 1982, 2026, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.31.0.0.0
SQL> ALTER SYSTEM STOP ROLLING MIGRATION;
System altered.
Node 2:
oracle@rac001_lab2:~/sqlplus "/as sysasm"
SQL*Plus: Release 19.0.0.0.0 - Production on Wed Nov 4 11:05:16 2026
Version 19.31.0.0.0
Copyright (c) 1982, 2026, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.31.0.0.0
SQL> ALTER SYSTEM STOP ROLLING MIGRATION;
System altered.
step 3.Run downgrade on the impacted node:
/u01/app/23.26.1/grid/crs/install/rootcrs.sh -downgrade
Using configuration parameter file: /u01/app/23.26.1/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/oracle/crsdata/rac001_lab/crsconfig/crsdowngrade_rac001_lab_2026-11-04_09-23-25AM.log
2026/11/04 11:24:07 CLSRSC-4006: Removing Oracle Autonomous Health Framework (AHF).
2026/11/04 11:24:39 CLSRSC-4007: Successfully removed Oracle Autonomous Health Framework (AHF).
2026/11/04 11:24:42 CLSRSC-591: successfully downgraded Oracle Clusterware stack on this node
2026/11/04 11:24:46 CLSRSC-592: Run 'crsctl start crs' from home /u01/app/19.31.0.0/grid on each node to complete downgrade.
step 4. Start crs from old crs home and check configuration
Node 1
/u01/app/19.31.0.0/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
/u01/app/19.31.0.0/grid/bin/crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [19.31.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3367946646].
Oracle Clusterware version on node [rac001_lab] is [19.31.0.0]
Node 2
/u01/app/19.31.0.0/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
/u01/app/19.31.0.0/grid/bin/crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [19.31.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3367946646].
Oracle Clusterware version on node [rac001_lab2] is [19.31.0.0]
Now you get back to NORMAL State from before broken upgrade.
To not repeat my mistake, download and install newer version of Clusterware release update in my case issue didn’t happens on 23.26.4 upgrade.