Introduction
Patching your Oracle Database Appliance from 19.20 or earlier release to 19.21 or newer release implies the use of Data Preserving Reprovisioning (DPR). Most of the time, and with an adequate preparation, the DPR works fine. But if something goes wrong at the restore-node -g step, you will need some troubleshooting and maybe opening a SR. There is no possible rollback. Here is the problem I recently had when patching an ODA from 19.20 to 19.26 using the intermediate 19.24 for DPR.
What is DPR and why you need it on ODA?
Oracle Database Appliance has a global quarterly patch: this is the only patch you are supposed to use on this kind of hardware. The patch includes updates for Grid Infrastructure (GI) and database (DB), but also updates for the operating system (OS), BIOS, firmwares and other software components. Regarding the OS, Linux 7 was in use up to 19.20 version. Starting from 19.21, Linux 8 is mandatory and there is no update-server command available: you will need to reinstall the new OS on your ODA using DPR.
Basically, DPR is a complete reinstall of the OS without erasing the data disks. Before doing the reinstall, you will “detach” the node. After the reinstall, you will “restore” the node using an odacli command: restore-node -g. You can compare this to the unplug/plug operations on a PDB. Unplugging a PDB is writing the metadata in a file you will later use to plug in back your PDB somewhere else. The DPR feature is embedded in patch 19.21, 19.22, 19.23 and 19.24. As patches are only cumulative with the 4 previous versions, starting from 19.25 you’re supposed to use release 19.21 at least, meaning already in Linux 8.
My ODA setup: a real client environment
My client has 2x ODA X9-2S with Enterprise Edition and Data Guard for production databases. These ODAs were deployed 2 years ago, and already patched from 19.17 to 19.20 1 year ago. Now, it’s time to patch them to the latest 19.26, using DPR 19.24 for the Linux 8 jump.
Pre-upgrade report and detach-node
DCS components are updated to 19.24 at first step, then the pre-upgrade report can be tried:
odacli update-dcsadmin -v 19.24.0.0.0
odacli update-dcscomponents -v 19.24.0.0.0
odacli update-dcsagent -v 19.24.0.0.0
odacli create-preupgradereport -bm
odacli describe-preupgradereport -i 03f53c9c-fe82-4c2b-bf18-49fd31853054
...
This pre-upgrade report MUST be successful: if not, solve the listed problems and retry until it’s OK.
Once OK, the detach-node will backup the metadata of your ODA for later restore:
odacli detach-node -all
As detach files are stored locally, make sure to backup these files on an external volume otherwise you will not be able to restore your data after reimaging.
cp -r /opt/oracle/oak/restore/out/* /mnt/backup/ODA_backup/
Reimaging the ODA
Reimaging is done with patch 30403643 (version 19.24 in my case). This is an ISO file you will virtually plug in through ILOM as a virtual CDROM. Then you will reboot the server and automatic setup will start. After this reimaging, you will need to do a configure-firstnet for basic network configuration, and then register the new GI and the detached server archive:
odacli update-repository -f /mnt/backup/ODA_backup/odacli-dcs-19.24.0.0.0-240724-GI-19.24.0.0.zip
odacli update-repository -f /mnt/backup/ODA_backup/serverarchive_srvoda1.zipOnce both files have successfully been registered, the restore-node can be done:
odacli restore-node -g
PRGH-1030 and timeout of the restore-node -g
The restore-node -g is similar to a create-appliance: it will configure the system, create the users, provision GI and configure ASM. Instead of configuring ASM with fresh disks without any data, it will read the ASM headers on disks and mount the existing diskgroups. It means that you will get back the DB homes, the databases and ACFS volumes.
This time, the restore-node -g didn’t work for me, it took a very long time (more than one hour) before ending with a failure, a timeout was probably triggered as odacli jobs always finish with a success or a failure:
odacli describe-job -i "46addc5f-7a1f-4f4b-bccf-78cb3708bef9"
Job details
----------------------------------------------------------------
ID: 46addc5f-7a1f-4f4b-bccf-78cb3708bef9
Description: Restore node service - GI
Status: Failure (To view Error Correlation report, run "odacli describe-job -i 46addc5f-7a1f-4f4b-bccf-78cb3708bef9 --ecr" command)
Created: March 5, 2025 9:03:44 AM CET
Message: DCS-10001:Internal error encountered: Failed to provision GI with RHP at the home: /u01/app/19.24.0.0/grid: DCS-10001:Internal error encountered: PRGH-1030 : The environments on nodes 'srvoda1' do not satisfy some of the prerequisite checks.
..
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ---------
Restore node service creation March 5, 2025 9:04:01 AM CET March 5, 2025 10:17:35 AM CET Failure
Restore node service creation March 5, 2025 9:04:01 AM CET March 5, 2025 10:17:35 AM CET Failure
Restore node service creation March 5, 2025 9:04:01 AM CET March 5, 2025 10:17:35 AM CET Failure
Provisioning service creation March 5, 2025 9:04:01 AM CET March 5, 2025 10:17:35 AM CET Failure
Provisioning service creation March 5, 2025 9:04:01 AM CET March 5, 2025 10:17:35 AM CET Failure
Validate absence of Interconnect March 5, 2025 9:04:01 AM CET March 5, 2025 9:04:02 AM CET Success
network configuration file
Setting up Network March 5, 2025 9:04:03 AM CET March 5, 2025 9:04:03 AM CET Success
Restart network interface pubnet March 5, 2025 9:04:03 AM CET March 5, 2025 9:04:09 AM CET Success
Setting up Vlan March 5, 2025 9:04:09 AM CET March 5, 2025 9:04:10 AM CET Success
Restart network interface priv0.100 March 5, 2025 9:04:10 AM CET March 5, 2025 9:04:11 AM CET Success
Restart network interface privasm March 5, 2025 9:04:11 AM CET March 5, 2025 9:04:11 AM CET Success
Setting up Network March 5, 2025 9:04:11 AM CET March 5, 2025 9:04:11 AM CET Success
Restart network interface privasm March 5, 2025 9:04:11 AM CET March 5, 2025 9:04:17 AM CET Success
Network update March 5, 2025 9:04:17 AM CET March 5, 2025 9:04:30 AM CET Success
Updating network March 5, 2025 9:04:17 AM CET March 5, 2025 9:04:30 AM CET Success
Setting up Network March 5, 2025 9:04:17 AM CET March 5, 2025 9:04:17 AM CET Success
Restart network interface btbond1 March 5, 2025 9:04:17 AM CET March 5, 2025 9:04:20 AM CET Success
Restart network interface btbond1 March 5, 2025 9:04:20 AM CET March 5, 2025 9:04:24 AM CET Success
Restart network interface pubnet March 5, 2025 9:04:24 AM CET March 5, 2025 9:04:29 AM CET Success
Validate availability of pubnet March 5, 2025 9:04:30 AM CET March 5, 2025 9:04:30 AM CET Success
OS usergroup 'asmdba' creation March 5, 2025 9:04:30 AM CET March 5, 2025 9:04:30 AM CET Success
OS usergroup 'asmoper' creation March 5, 2025 9:04:30 AM CET March 5, 2025 9:04:30 AM CET Success
OS usergroup 'asmadmin' creation March 5, 2025 9:04:30 AM CET March 5, 2025 9:04:30 AM CET Success
OS usergroup 'dba' creation March 5, 2025 9:04:30 AM CET March 5, 2025 9:04:30 AM CET Success
OS usergroup 'dbaoper' creation March 5, 2025 9:04:30 AM CET March 5, 2025 9:04:30 AM CET Success
OS usergroup 'oinstall' creation March 5, 2025 9:04:30 AM CET March 5, 2025 9:04:31 AM CET Success
OS user 'grid' creation March 5, 2025 9:04:31 AM CET March 5, 2025 9:04:31 AM CET Success
OS user 'oracle' creation March 5, 2025 9:04:31 AM CET March 5, 2025 9:04:31 AM CET Success
Default backup policy creation March 5, 2025 9:04:31 AM CET March 5, 2025 9:04:31 AM CET Success
Backup Config name validation March 5, 2025 9:04:31 AM CET March 5, 2025 9:04:31 AM CET Success
Backup config metadata persist March 5, 2025 9:04:31 AM CET March 5, 2025 9:04:31 AM CET Success
Grant permission to RHP files March 5, 2025 9:04:31 AM CET March 5, 2025 9:04:31 AM CET Success
Add SYSNAME in Env March 5, 2025 9:04:32 AM CET March 5, 2025 9:04:32 AM CET Success
Install oracle-ahf March 5, 2025 9:04:32 AM CET March 5, 2025 9:06:13 AM CET Success
Stop DCS Admin March 5, 2025 9:06:16 AM CET March 5, 2025 9:06:16 AM CET Success
Generate mTLS certificates March 5, 2025 9:06:16 AM CET March 5, 2025 9:06:17 AM CET Success
Exporting Public Keys March 5, 2025 9:06:17 AM CET March 5, 2025 9:06:18 AM CET Success
Creating Trust Store March 5, 2025 9:06:18 AM CET March 5, 2025 9:06:22 AM CET Success
Update config files March 5, 2025 9:06:22 AM CET March 5, 2025 9:06:22 AM CET Success
Restart DCS Admin March 5, 2025 9:06:22 AM CET March 5, 2025 9:06:43 AM CET Success
Unzipping storage configuration files March 5, 2025 9:06:43 AM CET March 5, 2025 9:06:43 AM CET Success
Reloading multipath devices March 5, 2025 9:06:43 AM CET March 5, 2025 9:06:44 AM CET Success
Restart oakd March 5, 2025 9:06:44 AM CET March 5, 2025 9:06:55 AM CET Success
Restart oakd March 5, 2025 9:08:05 AM CET March 5, 2025 9:08:15 AM CET Success
Restore Quorum Disks March 5, 2025 9:08:15 AM CET March 5, 2025 9:08:16 AM CET Success
Creating GI home directories March 5, 2025 9:08:16 AM CET March 5, 2025 9:08:16 AM CET Success
Extract GI clone March 5, 2025 9:08:16 AM CET March 5, 2025 9:09:29 AM CET Success
Creating wallet for Root User March 5, 2025 9:09:29 AM CET March 5, 2025 9:09:33 AM CET Success
Creating wallet for ASM Client March 5, 2025 9:09:33 AM CET March 5, 2025 9:09:39 AM CET Success
Grid stack creation March 5, 2025 9:09:39 AM CET March 5, 2025 10:17:35 AM CET Failure
GI Restore with RHP March 5, 2025 9:09:39 AM CET March 5, 2025 10:17:35 AM CET FailureAccording to the error message, my system doesn’t satisfy the prerequisites checks.
Let’s check if error correlation may help:
odacli describe-job -i 46addc5f-7a1f-4f4b-bccf-78cb3708bef9 --ecr
ODA Assistant - Error Correlation report
----------------------------------------
Failed job ID: 46addc5f-7a1f-4f4b-bccf-78cb3708bef9
Description: Restore node service - GI
Start Time: 2025-03-05 09:04:01
End Time: 2025-03-05 10:17:35
EC report path: /opt/oracle/dcs/da/da_repo/46addc5f-7a1f-4f4b-bccf-78cb3708bef9.json
Failed Task Messages
--------------------
[Restore node service - GI] - DCS-10001:Internal error encountered: Failed to provision GI with RHP at the home: /u01/app/19.24.0.0/grid: DCS-10001:Internal error encountered: PRGH-1030 : The environments on nodes 'srvoda1' do not satisfy some of the prerequisite checks. ..
srvoda1 Log Messages
----------------------------
DCS Agent
~~~~~~~~~
Error Logs
==========
[Install oracle-ahf] - Trying to add string [ERROR] : Unable to switch to home directory
[Install oracle-ahf] - error is package tfa-oda is not installed
[GI Restore with RHP] - Calling rhp provGI
[GI Restore with RHP] - Task(id: TaskDcsJsonRpcExt_707, name: GI Restore with RHP) failed
[GI Restore with RHP] - .. d.subtasks.isempty=true d.status=Failure
[Grid stack creation] - ..
[Grid stack creation] - .. d.subtasks.isempty=false d.status=Failure
[Grid stack creation] - DCS-10001:Internal error encountered: PRGH-1030 : The environments on nodes 'srvoda1' do not satisfy some of the prerequisite checks.
Error code - DCS-10001
Cause: An internal error occurred.
Action: Contact Oracle Support for assistance.
[Grid stack creation] - .., output:
Warning Logs
============
[[ SEND-THREAD 77 ]] - [ [ SEND-THREAD 77 ] dcs0-priv:22001] Request failed: Operation: GET Host: dcs0-priv:22001 Path: /joblocks/46addc5f-7a1f-4f4b-bccf-78cb3708bef9 Data: null Status: 404
[Provisioning service creation] - dm_multipath module is not loaded, attempting to load it...
RHP
~~~
Error Logs
==========
[jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - [SiteFactoryImpl.fetchSite:136] EntityNotExistsException PRGR-110 : Repository object "SRVODA1" of type "SITE" does not exist.
[jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - [WorkingCopyOperationImpl.internalAddGI:18517] Expected: Site does not exist, will have to be created :PRGR-119 : Site "srvoda1" does not exist.
[jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - [RemoteFactoryImpl.getCRSHomeOfRemoteCluster:1377] COException: PRCZ-4001 : failed to execute command "/bin/cat" using the privileged execution plugin "odaexec" on nodes "srvoda1" within 120 seconds
[jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - PRCZ-2103 : Failed to execute command "/bin/cat" on node "srvoda1" as user "root". Detailed error:
[jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - /bin/cat: /etc/oracle/olr.loc: No such file or directory
[jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - [WorkingCopyOperationImpl.internalAddGI:18972] Exec Exception, expected here :PRGR-118 : Working copy "OraGrid192400" does not exist.
[jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - [OracleHomeImpl.addWCMarkerFile:4403] Ignore the exception to create RHP WC marker file : PRKH-1009 : CRS HOME must be defined in the environment or in the Oracle Cluster Registry
[jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - [ExecCommandNoUserEqImpl.runCmd:491] Final CompositeOperation exception: PRCC-1021 : One or more of the submitted commands did not execute successfully.
[jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - PRCC-1026 : Command "/u01/app/19.24.0.0/grid/gridSetup.sh -silent -responseFile /u01/app/19.24.0.0/grid//crs/install/rhpdata/grid7317919467403765800.rsp -ignorePrereq -J-Dskip.cvu.root.checks=true -J-Doracle.install.grid.validate.all=false oracle_install_crs_ODA_CONFIG=olite oracle_install_crs_ConfigureMgmtDB=false -J-Doracle.install.crs.skipGIMRDiskGroupSizeCheck=true oracle_install_asm_UseExistingDG=true -J-Doracle.install.grid.validate.CreateASMDiskGroup=false" submitted on node srvoda1 timed out after 4,000 seconds.
[jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - </OUTPUT>
[jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - [OperationAPIImpl.provGI:676] OperationAPIException: PRGH-1030 : The environments on nodes 'srvoda1' do not satisfy some of the prerequisite checks.
Release Notes
-------------
No matching results were found.
Documentation
-------------
1. Error in restore node process in Data Preserving Reprovisioning
Abstract - In the Data Preserving Reprovisioning process, during node restore, an error may be encountered.
Link - https://srvoda1.dbi-services.net:7093/docs/cmtrn/issues-with-oda-odacli.html#GUID-F1385628-9F87-4FEF-8D27-289A3ED459EC
2. Error in restore node process in Data Preserving Reprovisioning
Abstract - In the Data Preserving Reprovisioning process, during node restore, an error may be encountered.
Link - https://srvoda1.dbi-services.net:7093/docs/cmtrn/issues-with-oda-odacli.html#GUID-75D52887-D425-4753-AF44-EFAB5C148873
3. Managing Backup, Restore, and Recovery on a DB System in a KVM Deployment
Abstract - Understand the backup, restore, and recovery operations supported on a DB system in a KVM deployment.
Link - https://srvoda1.dbi-services.net:7093/docs/cmtxp/managing-oracle-database-appliance-kvm-deployment1.html#GUID-7318F4D7-4CB8-486C-9DC7-A7490925B866
4. Backup, Restore and Recover Databases
Abstract - Review backup, restore, and recovery operations for your databases.
Link - https://srvoda1.dbi-services.net:7093/docs/cmtxp/backup-recover-restore.html#GUID-032C43EC-20B9-4036-ADA9-7631EEBBFEF6
5. Using the CLI to Backup, Restore, and Recover
Abstract - Use the command-line interface to backup, restore, and recover databases.
Link - https://srvoda1.dbi-services.net:7093/docs/cmtxp/backup-recover-restore.html#GUID-54F9A4A6-59B8-4A18-BE41-1CCB9096E2C5
NOTE: For additional details such as file name and line numbers of error logs, please refer to /opt/oracle/dcs/da/da_repo/924d139e-50cc-4893-ab0a-6acb7e7eeb9c.json
There is a lot of errors, but hard to find what is the source problem.
Let’s have a look at the logfile of the GI setup:
vi /u01/app/oraInventory/logs/GridSetupActions2025-03-05_09-09-40AM/gridSetupActions2025-03-05_09-09-40AM.out
…
Resolution failed, cached value is left untouched for variable
…
No ECDSA host key is known for srvoda1 and you have requested strict checking
…Could it be a DNS issue? Regarding the network settings, I’m pretty sure that IP, netmask and gateway are OK.
Identify and solve the problem by editing the json file
Like the create-appliance, the restore-node -g will use a json file restored previously with the update-repository of the server archive.
Let’s compare my own backup files and the json file used by restore-node -g:
cat /mnt/backup/ODA_backup/backup_ODA_srvoda1_20250312_0820/resolv.conf
search dbi-services.net
nameserver 10.1.2.127
nameserver 10.1.2.128
cat /opt/oracle/oak/restore/metadata/provisionInstance.json | grep dns
"dnsServers" : [ "10.4.110.4", "10.4.110.5" ],OK, DNS servers have been changed manually in the resolv.conf. They are no more correct in the json file.
As the restore-node -g failed, a cleanup of the system is mandatory:
/opt/oracle/oak/onecmd/cleanup.pl
INFO: Log file is /opt/oracle/oak/log/srvoda1/cleanup/cleanup_2025-03-05_10-40-06.log
INFO: Log file is /opt/oracle/oak/log/srvoda1/cleanup/dcsemu_diag_precleanup_2025-03-05_10-40-06.log
INFO: Platform is 'BM'
INFO: *******************************************************************
INFO: ** Starting process to cleanup provisioned host srvoda1 **
INFO: *******************************************************************
WARNING: DPR environment detected. DPR specific cleanup involves
WARNING: deconfiguring the ODA software stack without touching ASM
WARNING: storage to allow rerunning of the 'odacli restore-node -g'
WARNING: command. If regular cleanup(which erases ASM disk headers)
WARNING: is intended, rerun cleanup.pl with '-nodpr' option.
WARNING: If Multi-User Access is enabled, use '-omausers' option to
WARNING: delete the custom users created during the previous run.
Do you want to continue (yes/no) : yes
INFO:
Running cleanup will delete Grid User - 'grid' and
INFO: DB user - 'oracle' and also the
INFO: groups 'oinstall,dba,asmadmin,asmoper,asmdba'
INFO: nodes will be rebooted
Do you want to continue (yes/no) : yes
INFO: /u01/app/19.24.0.0/grid/bin/crsctl.bin
INFO: *************************************
INFO: ** Checking for GI bits presence
INFO: *************************************
INFO: GI bits /u01/app/19.24.0.0/grid found on system under /u01/app directory...
INFO: *************************************
INFO: ** DPR Cleanup
INFO: *************************************
INFO: Nothing to do.
SUCCESS: DPR cleanup actions completed.
INFO: Attempting to stop DCS agent on local node
INFO: *************************************
INFO: ** Executing AFD cleanup commands
INFO: *************************************
INFO: *************************************
INFO: ** Cleaning Oracle HAMI for ODA
INFO: *************************************
INFO: ** - Oracle HAMI for ODA - ensembles cleaned successfully
INFO: ** - Oracle HAMI for ODA - users cleaned successfully
INFO: *************************************
INFO: ** Executing stack deinstall commands
INFO: *************************************
INFO: *************************************
INFO: ** Removing IPC objects
INFO: *************************************
INFO: Cleaning up IDM configurations...
Deleting directory </opt/oracle/dcs/idm>INFO: *************************************
INFO: ** Cleaning miscellaneous components:
INFO: *************************************
INFO: ** - reset limits.conf
INFO: ** - delete users
INFO: ** - delete groups
INFO: ** - hostname, gateway and hosts reset commands
INFO: ** - dcs cleanup and orphan files removal commands
INFO: Attempting to clean MySQL tables on local node
INFO: Cleaning up network bridges
INFO: default net is: pubnet
INFO: Cleaning up network bridges: pubnet on btbond1
INFO: Reset public interface: pubnet
INFO: Cleaning up network bridges: privasm on priv0.100
INFO: remove VLAN config: /etc/sysconfig/network-scripts/ifcfg-priv0.100
INFO: BaseDBCC cleanup - skip
INFO: *************************************
INFO: ** Removing KVM files
INFO: *************************************
INFO: *************************************
INFO: ** Removing BM CPU Pool files
INFO: *************************************
INFO: ** - networking cleaning commands
INFO: ** - UTC reset commands
INFO: *************************************
INFO: ** Removing Oracle AHF RPM
INFO: *************************************
INFO: Oracle AHF RPM is installed as : oracle-ahf-2405000-20240715121646.x86_64
INFO: Uninstalling Oracle AHF RPM
INFO: Oracle AHF RPM uninstalled successfully
INFO: Oracle AHF RPM is installed as : oracle-ahf-2405000-20240715121646.x86_64
INFO: Delete directory clones.local (if existing)...
INFO: Cleaning up ACFS mounts...
INFO: Reset password for 'root' to default value
INFO: Executing <command to reset root password to default value>
INFO: Removing SSH keys on srvoda1
INFO: Rebooting the system via <reboot>...
INFO: Executing <reboot>
INFO: Cleanup was successful
INFO: Log file is /opt/oracle/oak/log/srvoda1/cleanup/cleanup_2025-03-05_10-40-06.log
WARNING: After system reboot, please re-run "odacli update-repository" for GI/DB clones,
WARNING: before running "odacli restore-node -g".
Once the server has rebooted, let’s register again the GI and server archive files:
odacli update-repository -f /mnt/backup/ODA_backup/odacli-dcs-19.24.0.0.0-240724-GI-19.24.0.0.zip
odacli update-repository -f /mnt/backup/ODA_backup/serverarchive_srvoda1.zipLet’s change the DNS in the provisionInstance.json and do the restore-node -g:
sed -i 's/10.4.110.4/10.1.2.127/g' /opt/oracle/oak/restore/metadata/provisionInstance.json
sed -i 's/10.4.110.5/10.1.2.128/g' /opt/oracle/oak/restore/metadata/provisionInstance.json
cat /opt/oracle/oak/restore/metadata/provisionInstance.json | grep dns
"dnsServers" : [ "10.1.2.127", "10.1.2.128" ],
odacli restore-node -g
...
odacli describe-job -i "f2e14691-1fc4-4b8d-9186-cbb55a69c5dd"
Job details
----------------------------------------------------------------
ID: f2e14691-1fc4-4b8d-9186-cbb55a69c5dd
Description: Restore node service - GI
Status: Success
Created: March 5, 2025 5:33:33 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ---------
Restore node service creation March 5, 2025 5:33:40 PM CET March 5, 2025 5:53:33 PM CET Success
Validate absence of Interconnect March 5, 2025 5:33:40 PM CET March 5, 2025 5:33:41 PM CET Success
network configuration file
Setting up Network March 5, 2025 5:33:42 PM CET March 5, 2025 5:33:42 PM CET Success
Restart network interface pubnet March 5, 2025 5:33:42 PM CET March 5, 2025 5:33:48 PM CET Success
Setting up Vlan March 5, 2025 5:33:48 PM CET March 5, 2025 5:33:49 PM CET Success
Restart network interface priv0.100 March 5, 2025 5:33:49 PM CET March 5, 2025 5:33:50 PM CET Success
Restart network interface privasm March 5, 2025 5:33:50 PM CET March 5, 2025 5:33:50 PM CET Success
Setting up Network March 5, 2025 5:33:51 PM CET March 5, 2025 5:33:51 PM CET Success
Restart network interface privasm March 5, 2025 5:33:51 PM CET March 5, 2025 5:33:56 PM CET Success
Network update March 5, 2025 5:33:56 PM CET March 5, 2025 5:34:09 PM CET Success
Updating network March 5, 2025 5:33:56 PM CET March 5, 2025 5:34:09 PM CET Success
Setting up Network March 5, 2025 5:33:56 PM CET March 5, 2025 5:33:56 PM CET Success
Restart network interface btbond1 March 5, 2025 5:33:56 PM CET March 5, 2025 5:34:00 PM CET Success
Restart network interface btbond1 March 5, 2025 5:34:00 PM CET March 5, 2025 5:34:03 PM CET Success
Restart network interface pubnet March 5, 2025 5:34:03 PM CET March 5, 2025 5:34:09 PM CET Success
Validate availability of pubnet March 5, 2025 5:34:09 PM CET March 5, 2025 5:34:09 PM CET Success
OS usergroup 'asmdba' creation March 5, 2025 5:34:09 PM CET March 5, 2025 5:34:09 PM CET Success
OS usergroup 'asmoper' creation March 5, 2025 5:34:09 PM CET March 5, 2025 5:34:09 PM CET Success
OS usergroup 'asmadmin' creation March 5, 2025 5:34:09 PM CET March 5, 2025 5:34:09 PM CET Success
OS usergroup 'dba' creation March 5, 2025 5:34:09 PM CET March 5, 2025 5:34:10 PM CET Success
OS usergroup 'dbaoper' creation March 5, 2025 5:34:10 PM CET March 5, 2025 5:34:10 PM CET Success
OS usergroup 'oinstall' creation March 5, 2025 5:34:10 PM CET March 5, 2025 5:34:10 PM CET Success
OS user 'grid' creation March 5, 2025 5:34:10 PM CET March 5, 2025 5:34:10 PM CET Success
OS user 'oracle' creation March 5, 2025 5:34:10 PM CET March 5, 2025 5:34:11 PM CET Success
Default backup policy creation March 5, 2025 5:34:11 PM CET March 5, 2025 5:34:11 PM CET Success
Backup Config name validation March 5, 2025 5:34:11 PM CET March 5, 2025 5:34:11 PM CET Success
Backup config metadata persist March 5, 2025 5:34:11 PM CET March 5, 2025 5:34:11 PM CET Success
Grant permission to RHP files March 5, 2025 5:34:11 PM CET March 5, 2025 5:34:11 PM CET Success
Add SYSNAME in Env March 5, 2025 5:34:11 PM CET March 5, 2025 5:34:11 PM CET Success
Install oracle-ahf March 5, 2025 5:34:11 PM CET March 5, 2025 5:35:54 PM CET Success
Stop DCS Admin March 5, 2025 5:35:56 PM CET March 5, 2025 5:35:56 PM CET Success
Generate mTLS certificates March 5, 2025 5:35:56 PM CET March 5, 2025 5:35:58 PM CET Success
Exporting Public Keys March 5, 2025 5:35:58 PM CET March 5, 2025 5:35:59 PM CET Success
Creating Trust Store March 5, 2025 5:35:59 PM CET March 5, 2025 5:36:02 PM CET Success
Update config files March 5, 2025 5:36:02 PM CET March 5, 2025 5:36:02 PM CET Success
Restart DCS Admin March 5, 2025 5:36:02 PM CET March 5, 2025 5:36:23 PM CET Success
Unzipping storage configuration files March 5, 2025 5:36:23 PM CET March 5, 2025 5:36:23 PM CET Success
Reloading multipath devices March 5, 2025 5:36:23 PM CET March 5, 2025 5:36:24 PM CET Success
Restart oakd March 5, 2025 5:36:24 PM CET March 5, 2025 5:36:34 PM CET Success
Restart oakd March 5, 2025 5:37:45 PM CET March 5, 2025 5:37:55 PM CET Success
Restore Quorum Disks March 5, 2025 5:37:55 PM CET March 5, 2025 5:37:56 PM CET Success
Creating GI home directories March 5, 2025 5:37:56 PM CET March 5, 2025 5:37:56 PM CET Success
Extract GI clone March 5, 2025 5:37:56 PM CET March 5, 2025 5:39:07 PM CET Success
Creating wallet for Root User March 5, 2025 5:39:07 PM CET March 5, 2025 5:39:15 PM CET Success
Creating wallet for ASM Client March 5, 2025 5:39:15 PM CET March 5, 2025 5:39:18 PM CET Success
Grid stack creation March 5, 2025 5:39:18 PM CET March 5, 2025 5:49:35 PM CET Success
GI Restore with RHP March 5, 2025 5:39:18 PM CET March 5, 2025 5:46:26 PM CET Success
Updating GIHome version March 5, 2025 5:46:28 PM CET March 5, 2025 5:46:31 PM CET Success
Restarting Clusterware March 5, 2025 5:46:32 PM CET March 5, 2025 5:49:35 PM CET Success
Post cluster OAKD configuration March 5, 2025 5:49:35 PM CET March 5, 2025 5:50:30 PM CET Success
Mounting disk group DATA March 5, 2025 5:50:30 PM CET March 5, 2025 5:50:31 PM CET Success
Mounting disk group RECO March 5, 2025 5:50:38 PM CET March 5, 2025 5:50:45 PM CET Success
Setting ACL for disk groups March 5, 2025 5:50:53 PM CET March 5, 2025 5:50:55 PM CET Success
Register Scan and Vips to Public Network March 5, 2025 5:50:55 PM CET March 5, 2025 5:50:57 PM CET Success
Adding Volume DUMPS to Clusterware March 5, 2025 5:51:10 PM CET March 5, 2025 5:51:13 PM CET Success
Adding Volume ACFSCLONE to Clusterware March 5, 2025 5:51:13 PM CET March 5, 2025 5:51:15 PM CET Success
Adding Volume ODABASE_N0 to Clusterware March 5, 2025 5:51:15 PM CET March 5, 2025 5:51:18 PM CET Success
Adding Volume COMMONSTORE to Clusterware March 5, 2025 5:51:18 PM CET March 5, 2025 5:51:20 PM CET Success
Adding Volume ORAHOME_SH to Clusterware March 5, 2025 5:51:20 PM CET March 5, 2025 5:51:23 PM CET Success
Enabling Volume(s) March 5, 2025 5:51:23 PM CET March 5, 2025 5:52:26 PM CET Success
Discover ACFS clones config March 5, 2025 5:53:18 PM CET March 5, 2025 5:53:27 PM CET Success
Configure export clones resource March 5, 2025 5:53:26 PM CET March 5, 2025 5:53:27 PM CET Success
Discover DbHomes ACFS config March 5, 2025 5:53:27 PM CET March 5, 2025 5:53:30 PM CET Success
Discover OraHomeStorage volumes March 5, 2025 5:53:27 PM CET March 5, 2025 5:53:30 PM CET Success
Setting up Hugepages March 5, 2025 5:53:30 PM CET March 5, 2025 5:53:30 PM CET Success
Provisioning service creation March 5, 2025 5:53:32 PM CET March 5, 2025 5:53:32 PM CET Success
Persist new agent state entry March 5, 2025 5:53:32 PM CET March 5, 2025 5:53:32 PM CET Success
Persist new agent state entry March 5, 2025 5:53:32 PM CET March 5, 2025 5:53:32 PM CET Success
Restart DCS Agent March 5, 2025 5:53:32 PM CET March 5, 2025 5:53:33 PM CET SuccessThis time it worked fine. Next step will be the restore-node -d for restoring the databases.
The backup script I use before patching an ODA
Troubleshooting is easier if you can have a look at configuration files that were in use prior reimaging. Here is a script I’ve been using for years before patching or reimaging an ODA. I would recommend making your own script based on mine according to your specific configuration:
vi /mnt/backup/prepatch_backup.sh
# Backup important files before patching
export BKPPATH=/mnt/backup/ODA_backup/backup_ODA_`hostname`_`date +"%Y%m%d_%H%M"`
echo "Backing up to " $BKPPATH
mkdir -p $BKPPATH
odacli list-databases > $BKPPATH/list-databases.txt
ps -ef | grep pmon | grep -v ASM | grep -v APX | grep -v grep | cut -c 58- | sort > $BKPPATH/running-instances.txt
odacli list-dbhomes > $BKPPATH/list-dbhomes.txt
odacli list-dbsystems > $BKPPATH/list-dbsystems.txt
odacli list-vms > $BKPPATH/list-vms.txt
crontab -u oracle -l > $BKPPATH/crontab-oracle.txt
crontab -u grid -l > $BKPPATH/crontab-grid.txt
crontab -l > $BKPPATH/crontab-root.txt
cat /etc/fstab > $BKPPATH/fstab.txt
cat /etc/oratab > $BKPPATH/oratab.txt
cat /etc/sysconfig/network > $BKPPATH/etc-sysconfig-network.txt
cat /etc/hosts > $BKPPATH/hosts
cat /etc/resolv.conf > $BKPPATH/resolv.conf
cp /etc/krb5.conf $BKPPATH/
cp /etc/krb5.keytab $BKPPATH/
mkdir $BKPPATH/network-scripts
cp /etc/sysconfig/network-scripts/ifcfg* $BKPPATH/network-scripts/
odacli describe-system > $BKPPATH/describe-system.txt
odacli describe-component > $BKPPATH/describe-component.txt
HISTFILE=~/.bash_history
set -o history
history > $BKPPATH/history-root.txt
cp /home/oracle/.bash_history $BKPPATH/history-oracle.txt
df -h > $BKPPATH/filesystems-status.txt
for a in `odacli list-dbhomes -j | grep dbHomeLocation | awk -F '"' '{print $4}' | sort` ; do mkdir -p $BKPPATH/$a/network/admin/ ; cp $a/network/admin/tnsnames.ora $BKPPATH/$a/network/admin/; cp $a/network/admin/sqlnet.ora $BKPPATH/$a/network/admin/; done
for a in `odacli list-dbhomes -j | grep dbHomeLocation | awk -F '"' '{print $4}' | sort` ; do mkdir -p $BKPPATH/$a/owm/ ; cp -r $a/owm/* $BKPPATH/$a/owm/; done
cp `ps -ef | grep -v grep | grep LISTENER | awk -F ' ' '{print $8}' | awk -F 'bin' '{print $1}'`network/admin/listener.ora $BKPPATH/gridhome-listener.ora
cp `ps -ef | grep -v grep | grep LISTENER | awk -F ' ' '{print $8}' | awk -F 'bin' '{print $1}'`/network/admin/sqlnet.ora $BKPPATH/gridhome-sqlnet.ora
tar czf $BKPPATH/u01-app-oracle-admin.tgz /u01/app/oracle/admin/
tar czf $BKPPATH/u01-app-oracle-local.tgz /u01/app/oracle/local/
tar czf $BKPPATH/home.tgz /home/
cp /etc/passwd $BKPPATH/
cp /etc/group $BKPPATH/
echo "End"
echo "Backup files size:"
du -hs $BKPPATH
echo "Backup files content:"
ls -lrt $BKPPATH
Conclusion
I would recommend these 3 rules to save hours of troubleshooting:
- as mentioned in the ODA documentation: never change networks parameters manually (IP, gateway, hostname, DNS, bonding mode, …)
- document every manual change you make on your ODA (additional tools setup, specific settings, …)
- do an extensive backup of all configuration files before doing a patching or a DPR (use the script provided in the previous chapter for creating yours)
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/JDU_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/11/CBL_web-min-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/MAW_web-min-scaled.jpg)