Introduction
Oracle Database Appliance is still very popular today. Provisioning is fast: from unboxing to a first running database is done in one day. And everything is standardized.
Patching is not that straight forward. First, you can only apply the latest patch on top of the 4 previous ones, and if you come from an older release, you may have 2, 3 or more patches to apply to reach the current version. It could take hours, even days.
The question one could ask is if it wouldn’t be more efficient to do a full ODA reimaging instead of patching. Let’s try to find out.
Environment and upgrade plan
This is a “real” environment composed of 4x ODA X8-2M: 2x ODAs for production, 2x ODAs for test/development. These servers have been provisioned in 2020 with the first 19c production available on ODA: 19.6. They were updated twice in the past years to reach 19.20 in 2024. These 2 upgrades were not that easy for some reasons, and systems are not that clean in my opinion.
According to MOS note 2757884.1, these ODAs are supported until August 2027, meaning that the last patch for these servers will be provided at this time. My client wants to keep them for 2 more years at least. So it’s definitely worth it to apply the patch.
I could have applied patch 19.24 then patch 19.27 on these ODAs, but as the first jump requires a Data Preserving Reprovisioning (meaning an OS reimaging) and as my ODAs are not that clean, I discussed with my client and we chose to reimage the servers from scratch.
Note that the production ODAs we will focus on are running 15 Enterprise Edition databases, and that a Data Guard configuration exists for each of them.
The plan is to switch all primaries to one ODA, reimage the other one, wait 2 weeks to see if everything is fine, do the switchover to the freshly reimaged ODA and reimage the other one the same way. As I don’t want to patch my databases now, I will deploy a 19.20 DB home and migrate to the latest version later.
Note that when I started this reimaging, current patch was 19.27. In the meantime, 19.28 becomes available.
Before starting: check ODA’s health
Let’s first check if the ODA I planned to reimage is healthy. You should normally monitor the hardware alerts on your ODAs, and eventually configure Automatic Service Request (ASR) to address hardware issues as soon as possible.
odaadmcli show server | grep Problem
Open Problems : 0
Open Problems Report : System is healthyNo problem on this ODA. I wouldn’t do anything on an ODA that is not healthy: fix up hardware troubles before patching or reimaging an ODA.
Check if standby databases are synced and disable archive transport
Apart from DBTEST which is primary (I always have a standalone test database on my ODAs), every database must be a Physical Standby Database. Expected downtime for such a reimaging is several hours or days depending on the number of databases. So, checking if sync is OK is a good practice. I also disable transport to standby databases and step down Data Guard to Maximum Performance:
vi check_role.sql
conn / as sysdba
set pages 0
select name||' is here '|| database_role from v$database;
exit;
for a in `ps -ef | grep pmon | grep -v grep | grep -v ASM | grep -v APX | cut -c 58- | sort`; do . oraenv <<< $a 1> /dev/null ; sqlplus -s /nolog @check_role; done
BPAPRD is here PHYSICAL STANDBY
DBTEST is here PRIMARY
BMC3PRD is here PHYSICAL STANDBY
MLOGAR is here PHYSICAL STANDBY
NIMSHMP is here PHYSICAL STANDBY
NOVOTEC is here PHYSICAL STANDBY
NTRAMPRD is here PHYSICAL STANDBY
NWTASKP is here PHYSICAL STANDBY
OPERA is here PHYSICAL STANDBY
PBYPRD is here PHYSICAL STANDBY
PRAXPRD is here PHYSICAL STANDBY
RCAT is here PHYSICAL STANDBY
SMARTPRD is here PHYSICAL STANDBY
SMKPRD is here PHYSICAL STANDBY
SPORT1 is here PHYSICAL STANDBY
SPORT2 is here PHYSICAL STANDBY
for a in `ps -ef | grep pmon | grep -v grep | grep -v ASM | grep -v APX | grep -v DBTEST | cut -c 58- | sort`; do echo $a ; . oraenv <<< $a 1> /dev/null ; dgmgrl / "show configuration lag" | grep Lag; done
BPAPRD
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 1 second ago)
BMC3PRD
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
MLOGAR
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 1 second ago)
NIMSHMP
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
NOVOTEC
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
NTRAMPRD
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
NWTASKP
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
OPERA
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
PBYPRD
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
PRAXPRD
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 1 second ago)
RCAT
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 1 second ago)
SMARTPRD
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
SMKPRD
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 1 second ago)
SPORT1
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
SPORT2
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
for a in `ps -ef | grep pmon | grep -v grep | grep -v ASM | grep -v APX | cut -c 58- | sort`; do echo $a ; . oraenv <<< $a 1> /dev/null ; dgmgrl / "edit configuration set protection mode as MaxPerformance;" | grep Succeeded; dgmgrl / "EDIT DATABASE ${a}_IN SET STATE='TRANSPORT-OFF';" | grep Succeed ; done
BPAPRD
Succeeded.
Succeeded.
DBTEST
BMC3PRD
Succeeded.
Succeeded.
MLOGAR
Succeeded.
Succeeded
NIMSHMP
Succeeded.
Succeeded.
NOVOTEC
Succeeded.
Succeeded.
NTRAMPRD
Succeeded.
Succeeded
NWTASKP
Succeeded.
Succeeded.
OPERA
Succeeded.
Succeeded.
PBYPRD
Succeeded.
Succeeded.
PRAXPRD
Succeeded.
Succeeded.
RCAT
Succeeded.
Succeeded.
SMARTPRD
Succeeded.
Succeeded.
SMKPRD
Succeeded.
Succeeded.
SPORT1
Succeeded.
Succeeded.
SPORT2
Succeeded.
Succeeded.
Backup important files
As you will erase everything on this ODA, backup everything that needs to be backed up on a remote filesystem (/backup here). Usually, I use this script:
cat /backup/Patch19.27/dbi_prepatch_backup.sh
# Backup important files before patching
export BKPPATH=/backup/Patch19.27/backup_ODA_`hostname`_`date +"%Y%m%d_%H%M"`
echo "Backing up to " $BKPPATH
mkdir -p $BKPPATH
odacli list-databases > $BKPPATH/list-databases.txt
ps -ef | grep pmon | grep -v ASM | grep -v APX | grep -v grep | cut -c 58- | sort > $BKPPATH/running-instances.txt
odacli list-dbhomes > $BKPPATH/list-dbhomes.txt
odacli list-dbsystems > $BKPPATH/list-dbsystems.txt
odacli list-vms > $BKPPATH/list-vms.txt
crontab -u oracle -l > $BKPPATH/crontab-oracle.txt
crontab -u grid -l > $BKPPATH/crontab-grid.txt
crontab -l > $BKPPATH/crontab-root.txt
cat /etc/fstab > $BKPPATH/fstab.txt
cat /etc/oratab > $BKPPATH/oratab.txt
cat /etc/sysconfig/network > $BKPPATH/etc-sysconfig-network.txt
cat /etc/hosts > $BKPPATH/hosts
cat /etc/resolv.conf > $BKPPATH/resolv.conf
cat /etc/sysctl.conf > $BKPPATH/
cp /etc/krb5.conf $BKPPATH/
cp /etc/krb5.keytab $BKPPATH/
mkdir $BKPPATH/network-scripts
cp /etc/sysconfig/network-scripts/ifcfg* $BKPPATH/network-scripts/
odacli describe-system > $BKPPATH/describe-system.txt
odacli describe-component > $BKPPATH/describe-component.txt
HISTFILE=~/.bash_history
set -o history
history > $BKPPATH/history-root.txt
cp /home/oracle/.bash_history $BKPPATH/history-oracle.txt
df -h > $BKPPATH/filesystems-status.txt
for a in `odacli list-dbhomes -j | grep dbHomeLocation | awk -F '"' '{print $4}' | sort` ; do mkdir -p $BKPPATH/$a/network/admin/ ; cp $a/network/admin/tnsnames.ora $BKPPATH/$a/network/admin/; cp $a/network/admin/sqlnet.ora $BKPPATH/$a/network/admin/; done
for a in `odacli list-dbhomes -j | grep dbHomeLocation | awk -F '"' '{print $4}' | sort` ; do mkdir -p $BKPPATH/$a/owm/ ; cp -r $a/owm/* $BKPPATH/$a/owm/; done
cp `ps -ef | grep -v grep | grep LISTENER | awk -F ' ' '{print $8}' | awk -F 'bin' '{print $1}'`network/admin/listener.ora $BKPPATH/gridhome-listener.ora
cp `ps -ef | grep -v grep | grep LISTENER | awk -F ' ' '{print $8}' | awk -F 'bin' '{print $1}'`/network/admin/sqlnet.ora $BKPPATH/gridhome-sqlnet.ora
tar czf $BKPPATH/u01-app-oracle-admin.tgz /u01/app/oracle/admin/
tar czf $BKPPATH/u01-app-odaorabase-oracle-admin.tgz /u01/app/odaorabase/oracle/admin/
tar czf $BKPPATH/u01-app-oracle-local.tgz /u01/app/oracle/local/
tar czf $BKPPATH/home.tgz /home/
cp /etc/passwd $BKPPATH/
cp /etc/group $BKPPATH/
echo "End"
echo "Backup files size:"
du -hs $BKPPATH
echo "Backup files content:"
ls -lrt $BKPPATH
sh /backup/Patch19.27/dbi_prepatch_backup.sh
...This backup script is probably not extensive: take some time to make a suitable one for your needs.
Let’s also take a backup of the DB homes, in my case I only have one:
tar czf /backup/Patch19.27/odaprdcr_dbhome.tgz /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4
ls -lrth /backup/Patch19.27/odaprdcr_dbhome.tgz
-rw-r--r-- 1 root root 5.9G Sep 3 10:50 /backup/Patch19.27/odaprdcr_dbhome.tgzBackup standby spfiles and controlfiles
As this ODA is only running standby databases, I don’t need to back them up. I can use the backup from primary assuming a reliable backup strategy has been deployed on production’s databases. 2 files differ between primary and standby databases: spfile and controlfile. So let’s backup these 2 files on all databases, it just needs a couple of minutes for all:
su - oracle
mkdir /backup/Patch19.27/ctl_spf
cat /backup/Patch19.27/bck_ctl_spf.rcv
backup current controlfile format '/backup/Patch19.27/ctl_spf/ctl_%d_%T.bck';
backup spfile format '/backup/Patch19.27/ctl_spf/spf_%d_%T.bck';
exit;
for a in `ps -ef | grep pmon | grep -v grep | grep -v ASM | grep -v APX | grep -v DBTEST | cut -c 58- | sort`; do echo $a ; . oraenv <<< $a 1> /dev/null ; rman target / cmdfile="/backup/Patch19.27/bck_ctl_spf.rcv" | grep Succeeded; done
ls -lrt /backup/Patch19.27/ctl_spf
total 33592
-rw-r----- 1 oracle asmadmin 46104576 Sep 3 10:33 ctl_BPAPRD_20250903.bck
-rw-r----- 1 oracle asmadmin 114688 Sep 3 10:33 spf_BPAPRD_20250903.bck
-rw-r----- 1 oracle asmadmin 21233664 Sep 3 10:33 ctl_BMC3PRD_20250903.bck
-rw-r----- 1 oracle asmadmin 114688 Sep 3 10:33 spf_BMC3PRD_20250903.bck
-rw-r----- 1 oracle asmadmin 27656192 Sep 3 10:33 ctl_MLOGAR_20250903.bck
-rw-r----- 1 oracle asmadmin 114688 Sep 3 10:34 spf_MLOGAR_20250903.bck
-rw-r----- 1 oracle asmadmin 30965760 Sep 3 10:34 ctl_NIMSHMP_20250903.bck
-rw-r----- 1 oracle asmadmin 114688 Sep 3 10:34 spf_NIMSHMP_20250903.bck
-rw-r----- 1 oracle asmadmin 27394048 Sep 3 10:34 ctl_NOVOTEC_20250903.bck
-rw-r----- 1 oracle asmadmin 114688 Sep 3 10:34 spf_NOVOTEC_20250903.bck
-rw-r----- 1 oracle asmadmin 77955072 Sep 3 10:34 ctl_NTRAMPRD_20250903.bck
-rw-r----- 1 oracle asmadmin 114688 Sep 3 10:34 spf_NTRAMPRD_20250903.bck
-rw-r----- 1 oracle asmadmin 29294592 Sep 3 10:34 ctl_NWTASKP_20250903.bck
-rw-r----- 1 oracle asmadmin 114688 Sep 3 10:34 spf_NWTASKP_20250903.bck
-rw-r----- 1 oracle asmadmin 66387968 Sep 3 10:35 ctl_OPERA_20250903.bck
-rw-r----- 1 oracle asmadmin 114688 Sep 3 10:35 spf_OPERA_20250903.bck
-rw-r----- 1 oracle asmadmin 20905984 Sep 3 10:35 ctl_PBYPRD_20250903.bck
-rw-r----- 1 oracle asmadmin 114688 Sep 3 10:35 spf_PBYPRD_20250903.bck
-rw-r----- 1 oracle asmadmin 28868608 Sep 3 10:35 ctl_PRAXPRD_20250903.bck
-rw-r----- 1 oracle asmadmin 114688 Sep 3 10:35 spf_PRAXPRD_20250903.bck
-rw-r----- 1 oracle asmadmin 14352384 Sep 3 10:35 ctl_RCAT_20250903.bck
-rw-r----- 1 oracle asmadmin 114688 Sep 3 10:35 spf_RCAT_20250903.bck
-rw-r----- 1 oracle asmadmin 35028992 Sep 3 10:35 ctl_SMARTPRD_20250903.bck
-rw-r----- 1 oracle asmadmin 114688 Sep 3 10:35 spf_SMARTPRD_20250903.bck
-rw-r----- 1 oracle asmadmin 21594112 Sep 3 10:35 ctl_SMKPRD_20250903.bck
-rw-r----- 1 oracle asmadmin 114688 Sep 3 10:36 spf_SMKPRD_20250903.bck
-rw-r----- 1 oracle asmadmin 36798464 Sep 3 10:36 ctl_SPORT1_20250903.bck
-rw-r----- 1 oracle asmadmin 114688 Sep 3 10:36 spf_SPORT1_20250903.bck
-rw-r----- 1 oracle asmadmin 20840448 Sep 3 10:36 ctl_SPORT2_20250903.bck
-rw-r----- 1 oracle asmadmin 114688 Sep 3 10:36 spf_SPORT2_20250903.bckThe restore will be easier with these files.
Prepare the json file for ODA deployment
Before doing the cleanup, prepare the json file you will use for deploying again your ODA.
You can use the one used for initial deployment, or eventually create a new one.
You will need these things for the json:
- which edition you use: Standard Edition or Enterprise Edition
- hostname, IP, netmask and gateway of your server
- hostname, IP, netmask and gateway of your ILOM
- user and group ids if you don’t use the default ones
- ratio between DATA and RECO for disk partitioning (you may change from what was configured before)
- DNS and NTP IPs
- default’s network interface (usually btbond1)
- DBTEST parameters if you create a test database during provisioning (I would recommend doing so)
You will find most of these parameters in the backup you’ve done with the script.
Here is my json file:
{
"instance": {
"instanceBaseName": "odaprdcr",
"dbEdition": "EE",
"objectStoreCredentials": null,
"name": "odaprdcr",
"systemPassword": "*********",
"timeZone": "Europe/Zurich",
"domainName": "dbi-services.ch",
"ntpServers" : ["10.8.122.132","10.8.122.133"],
"dnsServers" : ["10.8.122.132","10.8.122.233"],
"isMultiUserAccessEnabled": "false",
"isRoleSeparated": true,
"osUserGroup": {
"users": [
{
"userName": "oracle",
"userRole": "oracleUser",
"userId": 1000
},
{
"userName": "grid",
"userRole": "gridUser",
"userId": 1001
}
],
"groups": [
{
"groupName": "oinstall",
"groupRole": "oinstall",
"groupId": 1001
},
{
"groupName": "dbaoper",
"groupRole": "dbaoper",
"groupId": 1002
},
{
"groupName": "dba",
"groupRole": "dba",
"groupId": 1000
},
{
"groupName": "asmadmin",
"groupRole": "asmadmin",
"groupId": 1004
},
{
"groupName": "asmoper",
"groupRole": "asmoper",
"groupId": 1005
},
{
"groupName": "asmdba",
"groupRole": "asmdba",
"groupId": 1006
}
]
}
},
"nodes": [
{
"nodeNumber": "0",
"nodeName": "odaprdcr",
"network": [
{
"ipAddress" : "10.192.100.103",
"subNetMask" : "255.255.255.0",
"gateway" : "10.192.100.254",
"nicName": "btbond1",
"networkType": [
"Public"
],
"isDefaultNetwork": true
}
],
"ilom": {
"ilomName": "odaprdcr-ilom",
"ipAddress":"10.192.200.103",
"subNetMask":"255.255.255.0",
"gateway":"10.192.200.254"
}
}
],
"grid": {
"vip": [],
"diskGroup": [
{
"diskGroupName": "DATA",
"diskPercentage": 85,
"redundancy": "NORMAL"
},
{
"diskGroupName": "RECO",
"diskPercentage": 15,
"redundancy": "NORMAL"
}
],
"language": "en",
"scan": null
},
"database": {
"dbName": "DBTEST",
"dbCharacterSet": {
"characterSet": "AL32UTF8",
"nlsCharacterset": "AL16UTF16",
"dbTerritory": "AMERICA",
"dbLanguage": "AMERICAN"
},
"dbRedundancy": null,
"dbRole": "PRIMARY",
"adminPassword": "*********",
"dbEdition": "EE",
"databaseUniqueName": "DBTEST",
"dbClass": "OLTP",
"dbVersion": "19.27.0.0.250415",
"dbHomeId": null,
"instanceOnly": false,
"isCdb": true,
"pdBName": "PDB1",
"dbShape": "odb1",
"pdbAdminuserName": "pdbadmin",
"enableTDE": false,
"dbType": "SI",
"dbStorage": "ASM",
"dbConsoleEnable": false,
"dbOnFlashStorage": false,
"backupConfigId": null,
"rmanBkupPassword": null,
"tdePassword": null
},
"acr": {
"acrEnable": false
},
"sysOraHomeStorage": {
"diskgroup": "DATA",
"volumeSize": 80
}
}
Cleanup the data disks
Cleanup of the data disk headers is mandatory before reimaging, otherwise provisioning will fail. Make sure you’re on the correct ODA, you can’t undo this action:
/opt/oracle/oak/onecmd/cleanup.pl
INFO: Log file is /opt/oracle/oak/log/odaprdcr/cleanup/cleanup_2025-09-03_10-53-04.log
INFO: Log file is /opt/oracle/oak/log/odaprdcr/cleanup/dcsemu_diag_precleanup_2025-09-03_10-53-04.log
INFO: *******************************************************************
INFO: ** Starting process to cleanup provisioned host odaprdcr **
INFO: *******************************************************************
INFO: Default mode being used to cleanup a provisioned system.
INFO: It will change all ASM disk status from MEMBER to FORMER
Do you want to continue (yes/no) : yes
INFO: nodes will be rebooted
Do you want to continue (yes/no) : yes
INFO: /u01/app/19.20.0.0/oracle/bin/crsctl.bin
INFO: *************************************
INFO: ** Checking for GI bits presence
INFO: *************************************
INFO: GI bits /u01/app/19.20.0.0/oracle found on system under /u01/app directory...
INFO: ** Disabling AFD filtering
SUCCESS: AFD filtering disabled on all devices
INFO: ** Saving disk partition info
INFO: *************************************
INFO: ** Executing Stop Clusterware commands
INFO: *************************************
INFO: *************************************
INFO: ** Remove Oracle Stack
…
INFO: Rebooting the system via <reboot>...
INFO: Executing <reboot>
INFO: Cleanup was successful
INFO: Log file is /opt/oracle/oak/log/odaprdcr/cleanup/cleanup_2025-09-03_10-53-04.log
WARNING: After system reboot, please re-run "odacli update-repository" for GI/DB clones,
WARNING: before running "odacli create-appliance".Reimage with the latest OS
ODA patch file 30403643 includes an ISO image you will use to reimage your ODA. From the ILOM, start the remote console, connect this ISO file, define next boot device as CDROM and do a power cycle of the server. Reimaging an ODA takes about 50 minutes and doesn’t need any input. Keep the remote console open for the next operation.
Configure firstnet
Once your ODA runs on a brand new OS, you will need to do the initial network setup through the remote console. So log into the console with root/welcome1, type configure-firstnet and provide IP address, netmask and gateway of your server. LACP mode will depend on your switch configuration, if you didn’t use it before, keep it disabled. On an ODA deployed in 2020, LACP mode was not available. You normally won’t use DHCP, neither VLAN as VLAN is usually tagged on the switch ports where your ODA is connected to.
configure-firstnet
...Once configured, you can reach your server through the network and copy the needed files for the next steps.
Patch the microcodes
Reimaging your ODA doesn’t mean your microcodes are updated, it’s only the software part that is new. Therefore, first task after reimaging is patching these microcodes.
Let’s copy and register the patch file on the ODA and apply the updates (patch 37817290 for 19.27):
cd /opt/dbi
unzip p37817290_1927000_Linux-x86-64.zip
odacli update-repository -f /opt/dbi/oda-sm-19.27.0.0.0-250601-server.zip
odacli describe-job -i "1e2dd483-2237-47b9-9c60-5395fa720fd2"
Job details
----------------------------------------------------------------
ID: 1e2dd483-2237-47b9-9c60-5395fa720fd2
Description: Repository Update
Status: Success
Created: September 03, 2025 09:53:24 UTC
Message: /opt/dbi/oda-sm-19.27.0.0.0-250601-server.zip
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- -------
Unzip bundle September 03, 2025 09:53:39 UTC September 03, 2025 09:53:54 UTC Success
odacli describe-component
System Version
--------------
19.27.0.0.0
System Node Name
----------------
oak
Local System Version
--------------------
19.27.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK 19.27.0.0.0 up-to-date
DCSCONTROLLER 19.27.0.0.0 up-to-date
DCSCLI 19.27.0.0.0 up-to-date
DCSAGENT 19.27.0.0.0 up-to-date
DCSADMIN 19.27.0.0.0 up-to-date
OS 8.10 up-to-date
ILOM 5.1.1.23.r151750 5.1.4.25.r160118
BIOS 52100600 52140100
LOCAL CONTROLLER FIRMWARE {
[c8, c9] 214.2.271.9 up-to-date
}
SHARED CONTROLLER FIRMWARE {
[c0, c1, c2, c3, c4, c5] VDV1RL06 up-to-date
}
LOCAL DISK FIRMWARE {
[c6d0, c6d1] XC311102 up-to-date
}
HMP 2.4.10.1.600 up-to-date
odacli create-prepatchreport -s -v 19.27.0.0.0
sleep 120 ; odacli describe-prepatchreport -i 8010302b-feca-4f23-92bc-6878aa586713
Prepatch Report
------------------------------------------------------------------------
Job ID: 8010302b-feca-4f23-92bc-6878aa586713
Description: Patch pre-checks for [OS, ILOM, SERVER] to 19.27.0.0.0
Status: SUCCESS
Created: September 3, 2025 9:54:56 AM UTC
Result: All pre-checks succeeded
Node Name
---------------
oak
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.27.0.0.0.
Is patch location available Success Patch location is available.
Verify OS patch Success There are no packages available for
an update
Validate command execution Success Skipped command execution verfication
- Instance is not provisioned
__ILOM__
Validate ILOM server reachable Success Successfully connected with ILOM
server using public IP and USB
interconnect
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.27.0.0.0.
Is patch location available Success Patch location is available.
Checking Ilom patch Version Success Successfully verified the versions
Patch location validation Success Successfully validated location
Validate command execution Success Skipped command execution verfication
- Instance is not provisioned
__SERVER__
Validate local patching Success Successfully validated server local
patching
Validate command execution Success Skipped command execution verfication
- Instance is not provisioned
odacli update-server -v 19.27.0.0.0
sleep 900 ; odacli describe-job -i 3a4ce5a7-6305-483d-a7c3-e6328d8a831e
Job details
----------------------------------------------------------------
ID: 3a4ce5a7-6305-483d-a7c3-e6328d8a831e
Description: Server Patching to 19.27.0.0.0
Status: Success
Created: September 03, 2025 09:57:34 UTC
Message:
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- -------
Validate ILOM server reachable September 03, 2025 09:57:36 UTC September 03, 2025 09:57:37 UTC Success
Stop DCS Admin September 03, 2025 09:57:38 UTC September 03, 2025 09:57:39 UTC Success
Generate mTLS certificates September 03, 2025 09:57:39 UTC September 03, 2025 09:57:40 UTC Success
Exporting Public Keys September 03, 2025 09:57:40 UTC September 03, 2025 09:57:41 UTC Success
Creating Trust Store September 03, 2025 09:57:41 UTC September 03, 2025 09:57:43 UTC Success
Restart DCS Admin September 03, 2025 09:57:43 UTC September 03, 2025 09:58:03 UTC Success
Update config files September 03, 2025 09:57:43 UTC September 03, 2025 09:57:43 UTC Success
Deactivate Unit[dnf-makecache.timer] September 03, 2025 09:58:04 UTC September 03, 2025 09:58:04 UTC Success
Deactivate Unit[kdump.service] September 03, 2025 09:58:04 UTC September 03, 2025 09:58:05 UTC Success
Modify BM udev rules September 03, 2025 09:58:05 UTC September 03, 2025 09:58:25 UTC Success
Creating repositories using yum September 03, 2025 09:58:25 UTC September 03, 2025 09:58:27 UTC Success
Updating YumPluginVersionLock rpm September 03, 2025 09:58:27 UTC September 03, 2025 09:58:29 UTC Success
Applying OS Patches September 03, 2025 09:58:29 UTC September 03, 2025 09:59:03 UTC Success
Creating repositories using yum September 03, 2025 09:59:03 UTC September 03, 2025 09:59:03 UTC Success
Applying HMP Patches September 03, 2025 09:59:04 UTC September 03, 2025 09:59:06 UTC Success
Applying Firmware local Disk Patches September 03, 2025 09:59:07 UTC September 03, 2025 09:59:10 UTC Success
Oda-hw-mgmt upgrade September 03, 2025 09:59:07 UTC September 03, 2025 09:59:07 UTC Success
Patch location validation September 03, 2025 09:59:07 UTC September 03, 2025 09:59:07 UTC Success
Applying Firmware local Controller Patch September 03, 2025 09:59:10 UTC September 03, 2025 09:59:15 UTC Success
Checking Ilom patch Version September 03, 2025 09:59:15 UTC September 03, 2025 09:59:15 UTC Success
Patch location validation September 03, 2025 09:59:15 UTC September 03, 2025 09:59:15 UTC Success
Save password in Wallet September 03, 2025 09:59:15 UTC September 03, 2025 09:59:16 UTC Success
Apply Ilom patch September 03, 2025 09:59:16 UTC September 03, 2025 10:09:24 UTC Success
Disabling IPMI v2 September 03, 2025 09:59:16 UTC September 03, 2025 09:59:16 UTC Success
Cleanup JRE Home September 03, 2025 10:09:24 UTC September 03, 2025 10:09:24 UTC Success
Copying Flash Bios to Temp location September 03, 2025 10:09:24 UTC September 03, 2025 10:09:24 UTC Success
Generating and saving BOM September 03, 2025 10:09:24 UTC September 03, 2025 10:09:27 UTC Success
Update System version September 03, 2025 10:09:24 UTC September 03, 2025 10:09:24 UTC Success
PreRebootNode Actions September 03, 2025 10:09:27 UTC September 03, 2025 10:09:27 UTC Success
Reboot Node September 03, 2025 10:09:27 UTC September 03, 2025 10:09:27 UTC Success
The system will reboot once patched, let’s check the versions:
odacli describe-component
System Version
--------------
19.27.0.0.0
System Node Name
----------------
oak
Local System Version
--------------------
19.27.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK 19.27.0.0.0 up-to-date
DCSCONTROLLER 19.27.0.0.0 up-to-date
DCSCLI 19.27.0.0.0 up-to-date
DCSAGENT 19.27.0.0.0 up-to-date
DCSADMIN 19.27.0.0.0 up-to-date
OS 8.10 up-to-date
ILOM 5.1.4.25.r160118 up-to-date
BIOS 52140100 up-to-date
LOCAL CONTROLLER FIRMWARE {
[c8, c9] 214.2.271.9 up-to-date
}
SHARED CONTROLLER FIRMWARE {
[c0, c1, c2, c3, c4, c5] VDV1RL06 up-to-date
}
LOCAL DISK FIRMWARE {
[c6d0, c6d1] XC311102 up-to-date
}
HMP 2.4.10.1.600 up-to-date
No need to apply the storage patch here, but if needed, apply it with:
odacli update-storage -v 19.27.0.0.0
Provision the appliance
Now it’s time to put back Oracle binaries on this system. First register the GI and DB clones, then do the create-appliance with the json file you prepared earlier:
cd /opt/dbi
unzip p30403662_1927000_Linux-x86-64.zip
unzip p30403673_1927000_Linux-x86-64.zip
odacli update-repository -f /opt/dbi/odacli-dcs-19.27.0.0.0-250527-GI-19.27.0.0.zip
sleep 90 ; odacli describe-job -i c8f3559f-7f93-4df1-ba99-7511111f27ce
Job details
----------------------------------------------------------------
ID: c8f3559f-7f93-4df1-ba99-7511111f27ce
Description: Repository Update
Status: Success
Created: September 03, 2025 10:19:04 UTC
Message: /opt/dbi/odacli-dcs-19.27.0.0.0-250527-GI-19.27.0.0.zip
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- -------
Unzip bundle September 03, 2025 10:19:04 UTC September 03, 2025 10:20:08 UTC Success
odacli update-repository -f /opt/dbi/odacli-dcs-19.27.0.0.0-250527-DB-19.27.0.0.zip
sleep 90 ; odacli describe-job -i 96187596-16f6-413c-8905-9d85c6d3e106
Job details
----------------------------------------------------------------
ID: 96187596-16f6-413c-8905-9d85c6d3e106
Description: Repository Update
Status: Success
Created: September 03, 2025 10:20:50 UTC
Message: /opt/dbi/odacli-dcs-19.27.0.0.0-250527-DB-19.27.0.0.zip
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- -------
Unzip bundle September 03, 2025 10:20:50 UTC September 03, 2025 10:21:46 UTC Success
odacli create-appliance -r /opt/dbi/deploy_odaprdcr.txt
Enter an initial password for Web Console account (oda-admin): **********
Confirm the password for Web Console account (oda-admin): **********
User 'oda-admin' created successfully...
odacli describe-job -i 22fec8d6-d9ae-4576-8f18-d8e1a0b00370
Job details
----------------------------------------------------------------
ID: 22fec8d6-d9ae-4576-8f18-d8e1a0b00370
Description: Provisioning service creation
Status: Success
Created: September 03, 2025 12:45:37 CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- -------
Validate absence of Interconnect September 03, 2025 12:45:40 CEST September 03, 2025 12:45:40 CEST Success
network configuration file
Restart network interface pubnet September 03, 2025 12:45:43 CEST September 03, 2025 12:45:49 CEST Success
Setting up Network September 03, 2025 12:45:43 CEST September 03, 2025 12:45:43 CEST Success
Setting up Vlan September 03, 2025 12:45:50 CEST September 03, 2025 12:45:51 CEST Success
Restart network interface priv0.100 September 03, 2025 12:45:51 CEST September 03, 2025 12:45:52 CEST Success
Restart network interface privasm September 03, 2025 12:45:52 CEST September 03, 2025 12:45:53 CEST Success
Restart network interface privasm September 03, 2025 12:45:53 CEST September 03, 2025 12:45:59 CEST Success
Setting up Network September 03, 2025 12:45:53 CEST September 03, 2025 12:45:53 CEST Success
Network update September 03, 2025 12:45:59 CEST September 03, 2025 12:46:13 CEST Success
Restart network interface btbond1 September 03, 2025 12:45:59 CEST September 03, 2025 12:46:03 CEST Success
Setting up Network September 03, 2025 12:45:59 CEST September 03, 2025 12:45:59 CEST Success
Updating network September 03, 2025 12:45:59 CEST September 03, 2025 12:46:13 CEST Success
Restart network interface btbond1 September 03, 2025 12:46:03 CEST September 03, 2025 12:46:07 CEST Success
Restart network interface pubnet September 03, 2025 12:46:07 CEST September 03, 2025 12:46:13 CEST Success
OS usergroup 'asmadmin' creation September 03, 2025 12:46:13 CEST September 03, 2025 12:46:13 CEST Success
OS usergroup 'asmdba' creation September 03, 2025 12:46:13 CEST September 03, 2025 12:46:13 CEST Success
OS usergroup 'asmoper' creation September 03, 2025 12:46:13 CEST September 03, 2025 12:46:13 CEST Success
OS usergroup 'dba' creation September 03, 2025 12:46:13 CEST September 03, 2025 12:46:14 CEST Success
Validate availability of pubnet September 03, 2025 12:46:13 CEST September 03, 2025 12:46:13 CEST Success
OS user 'grid' creation September 03, 2025 12:46:14 CEST September 03, 2025 12:46:15 CEST Success
OS usergroup 'dbaoper' creation September 03, 2025 12:46:14 CEST September 03, 2025 12:46:14 CEST Success
OS usergroup 'oinstall' creation September 03, 2025 12:46:14 CEST September 03, 2025 12:46:14 CEST Success
OS user 'oracle' creation September 03, 2025 12:46:15 CEST September 03, 2025 12:46:16 CEST Success
Add SYSNAME in Env September 03, 2025 12:46:16 CEST September 03, 2025 12:46:16 CEST Success
Backup Config name validation September 03, 2025 12:46:16 CEST September 03, 2025 12:46:16 CEST Success
Backup config metadata persist September 03, 2025 12:46:16 CEST September 03, 2025 12:46:16 CEST Success
Default backup policy creation September 03, 2025 12:46:16 CEST September 03, 2025 12:46:16 CEST Success
Grant permission to RHP files September 03, 2025 12:46:16 CEST September 03, 2025 12:46:16 CEST Success
Install oracle-ahf September 03, 2025 12:46:16 CEST September 03, 2025 12:47:23 CEST Success
Stop DCS Admin September 03, 2025 12:48:25 CEST September 03, 2025 12:48:26 CEST Success
Generate mTLS certificates September 03, 2025 12:48:26 CEST September 03, 2025 12:48:27 CEST Success
Exporting Public Keys September 03, 2025 12:48:27 CEST September 03, 2025 12:48:28 CEST Success
Creating Trust Store September 03, 2025 12:48:29 CEST September 03, 2025 12:48:31 CEST Success
Restart DCS Admin September 03, 2025 12:48:31 CEST September 03, 2025 12:48:52 CEST Success
Update config files September 03, 2025 12:48:31 CEST September 03, 2025 12:48:31 CEST Success
Creating GI home directories September 03, 2025 12:48:52 CEST September 03, 2025 12:48:52 CEST Success
Extract GI clone September 03, 2025 12:48:52 CEST September 03, 2025 12:50:11 CEST Success
Grid home creation September 03, 2025 12:48:52 CEST September 03, 2025 12:50:13 CEST Success
Storage discovery September 03, 2025 12:50:13 CEST September 03, 2025 12:51:33 CEST Success
Creating wallet for Root User September 03, 2025 12:51:33 CEST September 03, 2025 12:51:36 CEST Success
Creating wallet for ASM Client September 03, 2025 12:51:36 CEST September 03, 2025 12:51:39 CEST Success
Grid stack creation September 03, 2025 12:51:39 CEST September 03, 2025 13:00:38 CEST Success
Provisioning GI with RHP September 03, 2025 12:51:39 CEST September 03, 2025 12:57:27 CEST Success
Set CRS ping target September 03, 2025 12:57:28 CEST September 03, 2025 12:57:29 CEST Success
Updating GI home version September 03, 2025 12:57:29 CEST September 03, 2025 12:57:34 CEST Success
Restarting Clusterware September 03, 2025 12:57:35 CEST September 03, 2025 13:00:38 CEST Success
Post cluster OAKD configuration September 03, 2025 13:00:38 CEST September 03, 2025 13:01:33 CEST Success
Disk group 'RECO' creation September 03, 2025 13:01:41 CEST September 03, 2025 13:01:52 CEST Success
Setting ACL for disk groups September 03, 2025 13:01:52 CEST September 03, 2025 13:01:55 CEST Success
Modify DB file attributes September 03, 2025 13:01:55 CEST September 03, 2025 13:02:03 CEST Success
Register Scan and Vips to Public Network September 03, 2025 13:02:03 CEST September 03, 2025 13:02:05 CEST Success
Configure export clones resource September 03, 2025 13:03:24 CEST September 03, 2025 13:03:24 CEST Success
Volume 'commonstore' creation September 03, 2025 13:03:24 CEST September 03, 2025 13:03:41 CEST Success
ACFS File system 'DATA' creation September 03, 2025 13:03:41 CEST September 03, 2025 13:04:03 CEST Success
Creating ACFS database home September 03, 2025 13:04:04 CEST September 03, 2025 13:05:22 CEST Success
Database home creation September 03, 2025 13:04:04 CEST September 03, 2025 13:07:55 CEST Success
Validating dbHome available space September 03, 2025 13:05:23 CEST September 03, 2025 13:05:23 CEST Success
Create required directories September 03, 2025 13:05:24 CEST September 03, 2025 13:05:24 CEST Success
Creating DbHome Directory September 03, 2025 13:05:24 CEST September 03, 2025 13:05:24 CEST Success
Extract DB clone September 03, 2025 13:05:24 CEST September 03, 2025 13:06:18 CEST Success
ProvDbHome by using RHP September 03, 2025 13:06:18 CEST September 03, 2025 13:07:33 CEST Success
Enable DB options September 03, 2025 13:07:33 CEST September 03, 2025 13:07:47 CEST Success
Creating wallet for DB Client September 03, 2025 13:07:51 CEST September 03, 2025 13:07:55 CEST Success
Database Creation by RHP September 03, 2025 13:07:56 CEST September 03, 2025 13:20:28 CEST Success
Provisioning service creation September 03, 2025 13:07:56 CEST September 03, 2025 13:23:47 CEST Success
Change permission for xdb wallet files September 03, 2025 13:20:28 CEST September 03, 2025 13:20:29 CEST Success
Add Startup Trigger to Open all PDBS September 03, 2025 13:20:29 CEST September 03, 2025 13:20:30 CEST Success
Place SnapshotCtrlFile in sharedLoc September 03, 2025 13:20:30 CEST September 03, 2025 13:20:33 CEST Success
SqlPatch upgrade September 03, 2025 13:22:08 CEST September 03, 2025 13:22:30 CEST Success
Running dbms_stats init_package September 03, 2025 13:22:30 CEST September 03, 2025 13:22:31 CEST Success
Set log_archive_dest for Database September 03, 2025 13:22:31 CEST September 03, 2025 13:22:34 CEST Success
Updating the Database version September 03, 2025 13:22:34 CEST September 03, 2025 13:22:38 CEST Success
Users tablespace creation September 03, 2025 13:23:47 CEST September 03, 2025 13:23:50 CEST Success
Clear all listeners from Database September 03, 2025 13:23:50 CEST September 03, 2025 13:23:50 CEST Success
Clear local and remote listeners from September 03, 2025 13:23:52 CEST September 03, 2025 13:23:53 CEST Success
Database
Copy Pwfile to Shared Storage September 03, 2025 13:23:53 CEST September 03, 2025 13:23:59 CEST Success
Persist new agent state entry September 03, 2025 13:24:02 CEST September 03, 2025 13:24:02 CEST Success
Persist new agent state entry September 03, 2025 13:24:02 CEST September 03, 2025 13:24:02 CEST Success
Provisioning service creation September 03, 2025 13:24:02 CEST September 03, 2025 13:24:02 CEST Success
Restart DCS Agent September 03, 2025 13:24:02 CEST September 03, 2025 13:24:03 CEST Success
Creating the appliance usually takes 40 to 50 minutes.
Check core reduction
Reimaging an ODA will keep the core reduction, but make sure it’s true with these commands:
odacli list-cpucores
Node Cores Modified Job Status
----- ------ ---------------------------------- ---------------
0 4 September 3, 2025 9:46:11 AM CEST CONFIGURED
lscpu | grep Core
Core(s) per socket: 2Here I have 4 enabled cores, meaning 2 processor licenses.
Setup older DB home
As I will bring back my Data Guard setup with 19.20 databases, I will need a 19.20 DB home. Let’s register the clone file and create the DB home. An increase of the diagnostic dest is probably needed, 10GB is way too small for me:
odacli list-dbhome-storages
=============================================================================================================
ID Node Description Disk Group Volume Size(GB) Status
-------------------------------------- ---- -------------------- ---------- ----------- ---------- ----------
bef3ae8a-96d3-44d4-b179-1e5dbdd2a939 0 ORACLE_HOME DATA orahome_sh 80 CONFIGURED
e972cc17-ff92-4203-b23f-55fab02a7cc0 0 ORACLE_BASE DATA odabase_n0 10 CONFIGURED
=============================================================================================================
odacli modify-dbhome-storage -i e972cc17-ff92-4203-b23f-55fab02a7cc0 -s 50
sleep 10 ; odacli list-dbhome-storages
=============================================================================================================
ID Node Description Disk Group Volume Size(GB) Status
-------------------------------------- ---- -------------------- ---------- ----------- ---------- ----------
bef3ae8a-96d3-44d4-b179-1e5dbdd2a939 0 ORACLE_HOME DATA orahome_sh 80 CONFIGURED
e972cc17-ff92-4203-b23f-55fab02a7cc0 0 ORACLE_BASE DATA odabase_n0 50 CONFIGURED
=============================================================================================================
cd /opt/dbi
unzip p30403662_1920000_Linux-x86-64.zip
odacli update-repository -f /opt/dbi/odacli-dcs-19.20.0.0.0-230720-DB-19.20.0.0.zip
sleep 30 ; odacli describe-job -i 3ee5db4d-d350-47b4-8e24-18949d244ab7
Job details
----------------------------------------------------------------
ID: 3ee5db4d-d350-47b4-8e24-18949d244ab7
Description: Repository Update
Status: Success
Created: September 03, 2025 13:47:23 CEST
Message: /opt/dbi/odacli-dcs-19.20.0.0.0-230720-DB-19.20.0.0.zip
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- -------
Unzip bundle September 03, 2025 13:47:23 CEST September 03, 2025 13:47:55 CEST Success
odacli create-dbhome -v 19.20.0.0.230718
sleep 240 ; odacli describe-job -i 3123544b-60c5-418a-82b5-c110dd68aafa
Job details
----------------------------------------------------------------
ID: 3123544b-60c5-418a-82b5-c110dd68aafa
Description: Database Home OraDB19000_home2 creation with version :19.20.0.0.230718
Status: Success
Created: September 03, 2025 13:53:21 CEST
Message: Create Database Home
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- -------
Setting up SSH equivalence September 03, 2025 13:53:32 CEST September 03, 2025 13:53:34 CEST Success
Setting up SSH equivalence September 03, 2025 13:53:34 CEST September 03, 2025 13:53:35 CEST Success
Creating ACFS database home September 03, 2025 13:53:35 CEST September 03, 2025 13:53:35 CEST Success
Validating dbHome available space September 03, 2025 13:53:35 CEST September 03, 2025 13:53:35 CEST Success
Create required directories September 03, 2025 13:53:36 CEST September 03, 2025 13:53:36 CEST Success
Creating DbHome Directory September 03, 2025 13:53:36 CEST September 03, 2025 13:53:36 CEST Success
Extract DB clone September 03, 2025 13:53:36 CEST September 03, 2025 13:54:23 CEST Success
ProvDbHome by using RHP September 03, 2025 13:54:23 CEST September 03, 2025 13:55:39 CEST Success
Enable DB options September 03, 2025 13:55:39 CEST September 03, 2025 13:55:54 CEST Success
Creating wallet for DB Client September 03, 2025 13:55:57 CEST September 03, 2025 13:55:58 CEST Success
odacli list-dbhomes
ID Name DB Version DB Edition Home Location Status
---------------------------------------- -------------------- -------------------- ---------- -------------------------------------------------------- ----------
19ee71be-07ae-4373-9bb6-31db8195e36c OraDB19000_home1 19.27.0.0.250415 EE /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 CONFIGURED
cd2dc372-3f95-40f1-917c-ee0793c26664 OraDB19000_home2 19.20.0.0.230718 EE /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2 CONFIGURED
Create the databases
Your ODA registry needs to know the databases, so I will create them before doing the restore. I don’t use the -io (instance only) of odacli for database creation as srvctl entries will be missing. So let’s create a new standalone database and then remove its files:
odacli create-database -cs AL32UTF8 -no-c -u BMC3PRD_CR -dh 'cd2dc372-3f95-40f1-917c-ee0793c26664' -n BMC3PRD -s odb1s -l AMERICAN -dt AMERICA -no-co -r asm
sleep 660 ; odacli describe-job -i 40b80359-0607-4330-ae6e-055d284a780e
Job details
----------------------------------------------------------------
ID: 40b80359-0607-4330-ae6e-055d284a780e
Description: Database service creation with DB name: BMC3PRD
Status: Success
Created: September 03, 2025 15:11:48 CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- -------
Database Creation by RHP September 03, 2025 15:11:50 CEST September 03, 2025 15:19:24 CEST Success
Database Service creation September 03, 2025 15:11:50 CEST September 03, 2025 15:22:19 CEST Success
Setting up SSH equivalence September 03, 2025 15:11:50 CEST September 03, 2025 15:11:50 CEST Success
Setting up SSH equivalence September 03, 2025 15:11:50 CEST September 03, 2025 15:11:50 CEST Success
Change permission for xdb wallet files September 03, 2025 15:19:24 CEST September 03, 2025 15:19:25 CEST Success
Place SnapshotCtrlFile in sharedLoc September 03, 2025 15:19:25 CEST September 03, 2025 15:19:28 CEST Success
SqlPatch upgrade September 03, 2025 15:20:43 CEST September 03, 2025 15:20:50 CEST Success
Running dbms_stats init_package September 03, 2025 15:20:50 CEST September 03, 2025 15:20:51 CEST Success
Set log_archive_dest for Database September 03, 2025 15:20:51 CEST September 03, 2025 15:20:54 CEST Success
Updating the Database version September 03, 2025 15:20:54 CEST September 03, 2025 15:20:58 CEST Success
Create Users tablespace September 03, 2025 15:22:19 CEST September 03, 2025 15:22:20 CEST Success
Clear all listeners from Database September 03, 2025 15:22:20 CEST September 03, 2025 15:22:21 CEST Success
Copy Pwfile to Shared Storage September 03, 2025 15:22:23 CEST September 03, 2025 15:22:27 CEST Success
su - oracle
. oraenv <<< BMC3PRD
sqlplus / as sysdba
set pages 0
set feedback off
spool /tmp/rmfiles.sh
select 'asmcmd rm -fr '||file_name from dba_data_files;
select 'asmcmd rm -fr '||file_name from dba_temp_files;
select 'asmcmd rm -fr '||member from v$logfile;
select 'asmcmd rm -fr '||name from v$controlfile;
select 'asmcmd rm -fr '||value from v$parameter where name='spfile';
spool off
shutdown abort;
exit
exit
su - grid
sh /tmp/rmfiles.sh
exit
Do the same for the other databases. Don’t try to create your databases with multiple odacli jobs running at the same time: this is something ODA cannot do in parallel, creation will be serialized and jobs will be waiting for the current one to complete.
Restore the databases
I’m used to move the spfile from ASM to the filesystem, so it’s the exact same path as the other server. In the same time, I change the role of the database as odacli created it as a primary:
su - oracle
. oraenv <<< BMC3PRD
srvctl modify database -db BMC3PRD_CR -spfile /u01/app/oracle/admin/BMC3PRD/pfile/spfileBMC3PRD.ora -startoption mount -role PHYSICAL_STANDBYNow let’s restore the spfile and the controlfile. RMAN is able to start an instance without any spfile for recovering the spfile, it’s very convenient:
rman target /
startup nomount;
restore spfile from '/backup/Patch19.27/ctl_spf/spf_BMC3PRD_20250903.bck';
Starting restore at 04-SEP-25
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=433 device type=DISK
channel ORA_DISK_1: restoring spfile from AUTOBACKUP /backup/Patch19.27/ctl_spf/spf_BMC3PRD_20250903.bck
channel ORA_DISK_1: SPFILE restore from AUTOBACKUP complete
Finished restore at 04-SEP-25
startup force nomount;
restore controlfile from '/backup/Patch19.27/ctl_spf/ctl_BMC3PRD_20250903.bck';
Starting restore at 04-SEP-25
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=26 device type=DISK
channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:05
output file name=+RECO/BMC3PRD_CR/CONTROLFILE/current.486.1210937913
Finished restore at 04-SEP-25Now I can start restoring the database. Choose the parallel degree carefully: as your ODA has nothing else to do, I would recommend using the maximum channels you could for big databases, for example between 4 and 8 channels for an ODA with 4 enabled cores:
startup force mount;
run {
allocate channel c1 device type disk;
allocate channel c2 device type disk;
allocate channel c3 device type disk;
allocate channel c4 device type disk;
restore database ;
}
…
channel c4: restoring datafile 00004 to +DATA/BMC3PRD/DATAFILE/undotbs1.291.1042201603
channel c4: restoring datafile 00005 to +DATA/BMC3PRD/DATAFILE/users.294.1042202001
channel c4: reading from backup piece /backup/BMC3PRD/backupset/20250901_000002_inc0_BMC3PRD_45931446_s36914_p1.bck
channel c3: piece handle=/backup/BMC3PRD/backupset/20250901_000002_inc0_BMC3PRD_45931446_s36908_p2.bck tag=INC0_20250901_000002
channel c3: restored backup piece 2
channel c3: restore complete, elapsed time: 00:00:38
channel c1: piece handle=/backup/BMC3PRD/backupset/20250901_000002_inc0_BMC3PRD_45931446_s36911_p2.bck tag=INC0_20250901_000002
channel c1: restored backup piece 2
channel c1: restore complete, elapsed time: 00:00:42
channel c4: piece handle=/backup/BMC3PRD/backupset/20250901_000002_inc0_BMC3PRD_45931446_s36914_p1.bck tag=INC0_20250901_000002
channel c4: restored backup piece 1
channel c4: restore complete, elapsed time: 00:00:35
channel c2: piece handle=/backup/BMC3PRD/backupset/20250901_000002_inc0_BMC3PRD_45931446_s36911_p1.bck tag=INC0_20250901_000002
channel c2: restored backup piece 1
channel c2: restore complete, elapsed time: 00:00:54
Finished restore at 04-SEP-25
released channel: c1
released channel: c2
released channel: c3
released channel: c4
exitAdd TNS entries for Data Guard and a static entry for the listener
Your database is now restored, you’ll need to configure communication with the primary. Let’s add the following TNS entries:
vi $ORACLE_HOME/network/admin/tnsnames.ora
…
BMC3PRD_CR =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = odaprdcr)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = BMC3PRD_CR)
)
)
BMC3PRD_IN =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = odaprdin)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = BMC3PRD_IN)
)
)You will also need the static entry for your database in the listener.ora file:
su - grid
vi $ORACLE_HOME/network/admin/listener.ora
SID_LIST_LISTENER=
(SID_LIST=
(SID_DESC=
(GLOBAL_DBNAME = BMC3PRD_CR)
(ORACLE_HOME = /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2)
(SID_NAME = BMC3PRD)
)
)
exitCopy the password file and modify its location
Pick up the password file from primary server and relocate the file to the filesystem with srvctl:
scp oracle@odaprdin:/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2/dbs/orapwBMC3PRD /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2/dbs/
srvctl modify database -db BMC3PRD_CR -pwfile
srvctl stop database -db BMC3PRD_CR ; sleep 10 ; srvctl start database -db BMC3PRD_CR -o mount
You can use a symbolic link in the dbs subfolder if you want to put this password file alongside your spfile.
Recover the database
Restored datafiles are probably several hours old, so let’s do a recover on the standby from the primary service before configuring Data Guard:
rman target sys
recover database from service 'BMC3PRD_IN';
…
channel ORA_DISK_1: using network backup set from service BMC3PRD_IN
destination for restore of datafile 00007: +DATA/BMC3PRD_CR/DATAFILE/dm_mc3prd_docbase.477.1210938063
channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: using network backup set from service BMC3PRD_IN
destination for restore of datafile 00008: +DATA/BMC3PRD_CR/DATAFILE/dm_mc3prd_docbase.478.1210938063
channel ORA_DISK_1: restore complete, elapsed time: 00:00:08
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: using network backup set from service BMC3PRD_IN
destination for restore of datafile 00009: +DATA/BMC3PRD_CR/DATAFILE/dm_mc3prd_docbase.479.1210938067
channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: using network backup set from service BMC3PRD_IN
destination for restore of datafile 00010: +DATA/BMC3PRD_CR/DATAFILE/dm_mc3prd_docbase.480.1210938165
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07
starting media recovery
media recovery complete, elapsed time: 00:00:00
Finished recover at 04-SEP-25
exit
If archivelog files are missing or if recovery is not possible, it doesn’t mean that it’s a problem. Data Guard should be able to solve a small or bigger archive gap.
Rebuild Data Guard configuration
Instead of adding back my standby database to the existing Data Guard configuration, I will recreate this configuration. It’s a matter of seconds:
dgmgrl sys
*****
Remove configuration;
create configuration DTG as primary database is 'BMC3PRD_IN' connect identifier is 'BMC3PRD_IN';
add database 'BMC3PRD_CR' as connect identifier is 'BMC3PRD_CR';
enable configuration;
edit database 'BMC3PRD_CR' set property LogXptMode='SYNC';
edit database 'BMC3PRD_IN' set property LogXptMode='SYNC';
edit database 'BMC3PRD_CR' set property StandbyFileManagement='AUTO';
edit database 'BMC3PRD_IN' set property StandbyFileManagement='AUTO';
EDIT DATABASE 'BMC3PRD_CR' SET PROPERTY 'ArchiveLagTarget'=1200;
EDIT DATABASE 'BMC3PRD_IN' SET PROPERTY 'ArchiveLagTarget'=1200;
edit configuration set protection mode as maxavailability;Depending on the lag between your primary and this standby, it can takes several minutes to bring back the sync of both databases:
show configuration lag;
Configuration - dtg
Protection Mode: MaxAvailability
Members:
BMC3PRD_IN - Primary database
BMC3PRD_CR - Physical standby database
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 1 second ago)
Fast-Start Failover: Disabled
Configuration Status:
SUCCESS (status updated 6 seconds ago)This database is OK, let’s continue with the next ones.
Additional settings
If you made additional settings on your ODA, you’ll have to configure them again. It could be:
- monitoring
- additional network interfaces
- additional services
- mount points and shares
- firewall rules
- kerberos authentication
- compatibility modes in sqlnet.ora file
- …
Conclusion
With several multi-TB databases to restore, it took me less than 3 days per ODA for this operation. And these ODAs are now clean and ready for their second life.
Reimaging an ODA from scratch is a good solution as soon as a Data Guard setup allows the databases to run on another ODA for several days. And with a good preparation, the job is not that huge. And there is much less troubleshooting compared to patching. This is something to consider.
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/JDU_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/MOP_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/JEW_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/MAW_web-min-scaled.jpg)