Introduction
Patch 19.21 is now available for Oracle Database Appliance series. Let’s find out what’s new and how to apply this patch.
What’s new?
The most important new component is Linux 8. X10 series were already running on Linux 8 with a dedicated patch (19.20.0.1), but older ODAs were stuck with Linux 7 since years.
This version also brings latest 19c RU to database and grid homes (bug and security fixes). 23c is not yet available on ODA and 21.8 is proposed as DB System only. Note that 21c being an innovation release, its main purpose if for testing some 23c features.
Nothing really new apart from OS major update this time, but this update was a long awaited one as Linux 9 is already available since months.
Which ODA is compatible with this 19.21 release?
The newest ODAs X10-HA, X10-L and X10-S are supported, as well as X9, X8 and X7. And that’s it. X6 series and X5-2HA are no longer supported since last year. X7-2 series becoming rather old now, you should start thinking about renewal this year or in early 2025. I usually recommend keeping your ODA 7 years.
Is this patch a cumulative one?
On X10, this 19.21 can be applied on top of 19.20.0.1 for sure, as the OS is already Linux 8.
For X9/X8/X7, you can apply 19.21 starting from 19.16 and later, but there is a catch. Patch will not be applied on top of your system, but system will need a fresh reimaging using DPR (Data Preserving Reprovisioning). It means that internal disks will be erased, OS will be redeployed and you will loose your specific system configurations. DPR will first “unplug” your ODA configuration in a set of files, and then plug in back after reimaging is done. It also means that you will not be able to use ODABR as a safety backup of your filesystems. If you compare this DPR patching to classic patching, it means more risks because it’s not possible to revert to old OS state.
In my environment, I will use an ODA X8-2M running 19.20 with multiple DB homes, databases, DB Systems and VMs. I will not describe applying this patch on X10 series as I don’t have this kind of ODA.
Is there also a patch for my databases?
Only databases version 19c are supported, and this is OK because this is the only one you should use now.
If you absolutely need an older release, I would recommend configuring an application VM for that purpose, as described here, but it comes without any guarantee of any kind.
Download the patch, ISO and clone files
35938481 => the patch itself
30403673 => the GI clone needed for deploying newer GI version (mandatory)
30403662 => the DB clone for deploying new version of 19c
30403643 => ISO file for reimaging (mandatory if you have X9/X8/X7)
32451228 => System image for DB Systems (if you use them)
Be sure to choose the very latest 19.21 when downloading the clones and the ISO, download link from MOS will first propose older version 19.4.
Prepare the patching
The prerequisites are different compared to a classic patching, just because your current OS configuration will be wiped out:
- make sure you have a procedure to put back your specific system settings
- backup your specifics scripts/files to an external drive
- check that backup is external and OK for all your databases (bare metal and DB Systems)
- do a VM backup (vDisk backup) on an external drive
- plan a confortable downtime for patching (I would recommend 1+ day)
As ODABR is useless (system disks will be erased), I would recommend doing a backup of the most important files: tnsnames.ora and listener.ora, database list, network configuration files, crontabs, home directory of system users, aso. If something goes wrong, you may need to do a full reimaging, meaning restoring everything from the backups.
Version precheck
Start to check current version on all the components:
odacli describe-component
System Version
---------------
19.20.0.0.0
System node Name
---------------
dbi-oda1
Local System Version
---------------
19.20.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK
19.20.0.0.0 up-to-date
GI
19.20.0.0.230718 up-to-date
DB {
[OraDB19000_home2 [bsn01]]
19.19.0.0.230418 19.20.0.0.230718
[OraDB19000_home3 [DBITST,LTTST]]
19.20.0.0.230718 up-to-date
[OraDB19000_home4 [WS12C,MYBRCDB,DBWEI]]
19.20.0.0.230718 up-to-date
[OraDB19000_home6]
19.20.0.0.230718 up-to-date
[OraDB12201_home1]
12.2.0.1.220118 not-available
}
DCSCONTROLLER
19.20.0.0.0 up-to-date
DCSCLI
19.20.0.0.0 up-to-date
DCSAGENT
19.20.0.0.0 up-to-date
DCSADMIN
19.20.0.0.0 up-to-date
OS
7.9 up-to-date
ILOM
5.1.1.23.r151750 up-to-date
BIOS
52100600 up-to-date
LOCAL CONTROLLER FIRMWARE {
[c3]
80000681 up-to-date
[c4]
8000C470 up-to-date
}
SHARED CONTROLLER FIRMWARE
VDV1RL06 up-to-date
LOCAL DISK FIRMWARE
XC311132 up-to-date
SHARED DISK FIRMWARE
XC311132 up-to-date
HMP
2.4.9.0.601 up-to-dateList the DB homes, databases, DB Systems and VMs:
odacli list-dbhomes
ID Name DB Version DB Edition Home Location Status
---------------------------------------- -------------------- ---------------------------------------- ---------- --------------------------------------------- ----------
f5c8e73a-f7dd-4142-a6b9-e571d02a5298 OraDB19000_home2 19.19.0.0.230418 SE /u01/app/odaorahome/oracle/product/19.0.0 CONFIGURED
.0/dbhome_2
6359448d-fbcd-4063-a677-f061d6f4d196 OraDB19000_home3 19.20.0.0.230718 SE /u01/app/odaorahome/oracle/product/19.0.0 CONFIGURED
.0/dbhome_3
1a89c106-9c9d-4219-b116-6c84aef5f668 OraDB19000_home4 19.20.0.0.230718 EE /u01/app/odaorahome/oracle/product/19.0.0 CONFIGURED
.0/dbhome_4
cefeaca6-1d68-4f16-91a3-dc3de982cae1 OraDB19000_home6 19.20.0.0.230718 EE /u01/app/odaorahome/oracle/product/19.0.0 CONFIGURED
.0/dbhome_6
de47b591-4040-49f9-99e7-b854a6df1b14 OraDB12201_home1 12.2.0.1.220118 SE /u01/app/odaorahome/oracle/product/12.2.0 CONFIGURED
.1/dbhome_1
odacli list-databases
ID DB Name DB Type DB Version CDB Class Edition Shape Storage Status DB Home ID
---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
63f06b37-282d-4972-90cd-b60a1a3c853e DBITST SI 19.20.0.0.230718 true OLTP SE odb1 ASM CONFIGURED 6359448d-fbcd-4063-a677-f061d6f4d196
0f85bd3a-a63e-465c-9285-035a4d7242d7 WS12C SI 19.20.0.0.230718 false OLTP EE odb1 ACFS CONFIGURED 1a89c106-9c9d-4219-b116-6c84aef5f668
e490fe71-d35e-404f-b799-f8fdb388f21f TST23 SI 19.20.0.0.230718 false OLTP SE odb1 ACFS CONFIGURED 26794187-3e84-4fdd-b054-82bbd3e1b7fa
e723b4d7-05bf-4a78-874e-fcfd52b3765d MYBRCDB SI 19.20.0.0.230718 false OLTP EE odb1 ASM CONFIGURED 1a89c106-9c9d-4219-b116-6c84aef5f668
445b84b1-8c84-4a59-b302-fe96a380abe5 bsn01 SI 19.19.0.0.230418 true OLTP SE odb2 ASM CONFIGURED f5c8e73a-f7dd-4142-a6b9-e571d02a5298
a591d3b1-8c59-4edf-a1d4-b662903ccf5c DBWEI SI 19.20.0.0.230718 true OLTP EE odb1 ASM CONFIGURED 1a89c106-9c9d-4219-b116-6c84aef5f668
110d18aa-f084-4b7f-ac96-6734dddb6d54 LTTST SI 19.20.0.0.230718 false OLTP SE odb1 ACFS CONFIGURED 6359448d-fbcd-4063-a677-f061d6f4d196
odacli list-dbsystems
Name Shape Cores Memory GI version DB version Status Created Updated
-------------------- ---------- ----- ---------- ------------------ ------------------ --------------- ------------------------ ------------------------
testdbs02 odb2 2 16.00 GB 19.20.0.0.230718 N/A CONFIGURED 2024-01-04 13:47:20 CET 2024-01-04 14:20:50 CET
testdbs01 odb2 2 16.00 GB 19.20.0.0.230718 19.20.0.0.230718 CONFIGURED 2024-01-04 12:07:05 CET 2024-01-04 12:39:39 CET
odacli list-vms
Name VM Storage Current State Target State Created Updated
-------------------- -------------------- --------------- --------------- ------------------------ ------------------------
ubuntu02 VMstore ONLINE ONLINE 2023-12-07 15:06:57 CET 2023-12-07 15:06:57 CET
VMMAW01 VMstore ONLINE ONLINE 2023-12-07 14:51:32 CET 2023-12-07 14:51:32 CET
delphix VMstore ONLINE ONLINE 2024-01-03 15:49:57 CET 2024-01-04 10:38:07 CET
ubuntu01 VMstore ONLINE ONLINE 2023-12-07 15:05:54 CET 2023-12-07 15:05:54 CET
Update the DCS components
Before reimaging, an update of the DCS components is required for correct unplugging of the ODA system:
cd /opt/dbi
unzip p35938481_1921000_Linux-x86-64.zip
odacli update-repository -f /opt/dbi/oda-sm-19.21.0.0.0-231220-server.zip
sleep 30 ; odacli describe-job -i "91189839-e855-48ea-a6b1-7d9695da52a5"
Job details
----------------------------------------------------------------
ID: 91189839-e855-48ea-a6b1-7d9695da52a5
Description: Repository Update
Status: Success
Created: January 9, 2024 4:04:55 PM CET
Message: /opt/dbi/oda-sm-19.21.0.0.0-231220-server.zip
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle January 9, 2024 4:04:55 PM CET January 9, 2024 4:05:30 PM CET Success
odacli describe-component
System Version
---------------
19.20.0.0.0
System node Name
---------------
dbi-oda1
Local System Version
---------------
19.20.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK
19.20.0.0.0 19.21.0.0.0
GI
19.20.0.0.230718 19.21.0.0.231017
DB {
[OraDB19000_home2 [bsn01]]
19.19.0.0.230418 19.21.0.0.231017
[OraDB19000_home3 [DBITST,LTTST]]
19.20.0.0.230718 19.21.0.0.231017
[OraDB19000_home4 [WS12C,MYBRCDB,DBWEI]]
19.20.0.0.230718 19.21.0.0.231017
[OraDB19000_home6]
19.20.0.0.230718 19.21.0.0.231017
[OraDB12201_home1]
12.2.0.1.220118 not-available
}
DCSCONTROLLER
19.20.0.0.0 19.21.0.0.0
DCSCLI
19.20.0.0.0 19.21.0.0.0
DCSAGENT
19.20.0.0.0 19.21.0.0.0
DCSADMIN
19.20.0.0.0 19.21.0.0.0
OS
7.9 8.8
ILOM
5.1.1.23.r151750 5.1.3.20.r153596
BIOS
52100600 52110200
LOCAL CONTROLLER FIRMWARE {
[c3]
80000681 up-to-date
[c4]
8000C470 8000D9AB
}
SHARED CONTROLLER FIRMWARE
VDV1RL06 up-to-date
LOCAL DISK FIRMWARE
XC311132 up-to-date
SHARED DISK FIRMWARE
XC311132 up-to-date
HMP
2.4.9.0.601 2.4.9.1.601Let’s update the DCS components to 19.21:
odacli update-dcsadmin -v 19.21.0.0.0
sleep 30 ; odacli describe-job -i "e28f0d1f-aecf-4a5c-a1d9-c8dd4ab4a2ee"
Job details
----------------------------------------------------------------
ID: e28f0d1f-aecf-4a5c-a1d9-c8dd4ab4a2ee
Description: DcsAdmin patching
Status: Success
Created: January 9, 2024 4:10:12 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation January 9, 2024 4:10:12 PM CET January 9, 2024 4:10:12 PM CET Success
Dcs-admin upgrade January 9, 2024 4:10:12 PM CET January 9, 2024 4:10:18 PM CET Success
sleep 30 ; odacli update-dcscomponents -v 19.21.0.0.0
{
"jobId" : "7e7b3ca7-a23c-4fd6-aa33-67f8cc2afea2",
"status" : "Success",
"message" : " DCS-Agent shutdown is successful. Skipping MySQL upgrade on OL7 Metadata schema update is done. dcsagent RPM upgrade is successful. dcscli RPM upgrade is successful. dcscontroller RPM upgrade is successful. Successfully reset the Keystore password. HAMI is not enabled Skipped removing old Libs. Successfully ran setupAgentAuth.sh ",
"reports" : null,
"createTimestamp" : "January 09, 2024 16:11:44 PM CET",
"description" : "Update-dcscomponents job completed and is not part of Agent job list",
"updatedTime" : "January 09, 2024 16:12:50 PM CET"
}
odacli update-dcsagent -v 19.21.0.0.0
sleep 180 ; odacli describe-job -i "8a6af503-3512-43eb-aff1-290ef301f72b"
Job details
----------------------------------------------------------------
ID: 8a6af503-3512-43eb-aff1-290ef301f72b
Description: DcsAgent patching
Status: Success
Created: January 9, 2024 4:13:31 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------
Dcs-agent upgrade to version January 9, 2024 4:13:31 PM CET January 9, 2024 4:15:48 PM CET Success
19.21.0.0.0
Update System version January 9, 2024 4:15:48 PM CET January 9, 2024 4:15:48 PM CET Success
Create a pre-upgrade report
The pre-upgrade report will tell you if your ODA is OK for unplugging its configuration:
odacli create-preupgradereport -bm
odacli describe-preupgradereport -i 12e859fb-5d39-4390-a6a9-a490657727e3
Upgrade pre-check report
------------------------------------------------------------------------
Job ID: 12e859fb-5d39-4390-a6a9-a490657727e3
Description: Run pre-upgrade checks for Bare Metal
Status: FAILED
Created: January 9, 2024 4:35:06 PM CET
Result: One or more pre-checks failed for [DB, DBSYSTEMS]
Node Name
---------------
dbi-oda1
Check Status Message Action
------------------------------ -------- -------------------------------------- --------------------------------------
__GI__
Check presence of databases Success No additional database found None
not managed by ODA registered in CRS
Check custom filesystems Success All file systems are owned and used None
by OS users provisioned by ODA
Check presence of HAVIP Success No HAVIP resources found registered None
resources not managed by ODA in CRS
Check presence of export Success No EXPORT resources found registered None
resources not managed by ODA in CRS
__OS__
Check Required OS files Success All the required files are present None
Check Additional OS RPMs Success No RPMs outside of base ISO were None
found on the system
__STORAGE__
Check Required Storage files Success All the required files are present None
Validate OAK Disks Success All OAK disks are in valid state None
Validate ASM Disk Groups Success All ASM disk groups are in valid state None
Validate ASM Disks Success All ASM disks are in valid state None
Check Database Home Storage Success The volume(s) orahome_sh,odabase_n0 None
volumes state is CONFIGURED.
Check space under /opt Success Free space on /opt: 10441.96 MB is None
more than required space: 1188 MB
Check space in ASM disk Success Space required for creating local None
group(s) homes is present in ACFS database
home storage. Required: 0 GB
Available: 38 GB
__SYS__
Validate Hardware Type Success Current hardware is supported None
Validate ILOM interconnect Success ILOM interconnect is not enabled None
Validate System Version Success System version 19.21.0.0.0 is None
supported
Verify System Timezone Success Succesfully verified the time zone None
file
Verify Grid User Success Grid user is verified None
Verify Grid Version Success Oracle Grid Infrastructure is running None
on the '19.20.0.0.230718' version on
all nodes
Check Audit Files Success Local Audit files not found None
__DB__
Validate Database Status Success Database 'DBITST' is running and is None
in 'CONFIGURED' state
Validate Database Version Success Version '19.20.0.0.230718' for None
database 'DBITST' is supported
Validate Database Datapatch Success Database 'DBITST' is completely None
Application Status applied with datapatch
Validate TDE wallet presence Success Database 'DBITST' is not TDE enabled. None
Skipping TDE wallet presence check.
Validate Database Home Success Database home location check passed None
location for database DBITST
Validate Database Status Success Database 'WS12C' is running and is in None
'CONFIGURED' state
Validate Database Version Success Version '19.20.0.0.230718' for None
database 'WS12C' is supported
Validate Database Datapatch Success Database 'WS12C' is completely None
Application Status applied with datapatch
Validate TDE wallet presence Success Database 'WS12C' is not TDE enabled. None
Skipping TDE wallet presence check.
Validate Database Home Success Database home location check passed None
location for database WS12C_SITE1
Validate Database Status Failed DCS-10001:Internal error encountered: Please check dcs-agent logs
Did not find the dbhome.
Validate Database Version Success Version '19.20.0.0.230718' for None
database 'TST23' is supported
Validate Database Datapatch Failed Error occurred while checking Fix the issue and then rerun
Application Status datapatch application status for create-preupgradereport' command
Database 'TST23': DCS-10001:Internal
error encountered: Database TST23 is
not registered with clusterware.
Validate TDE wallet presence Failed DCS-10001:Internal error encountered: Please check dcs-agent logs
Database TST23 is not registered with
clusterware.
Validate Database Home Failed ORACLE_HOME for Database TST23 is not Check why the database home is not
location set in CRS set in CRS. Please start up the
database if needed.
Validate Database Status Success Database 'MYBRCDB' is running and is None
in 'CONFIGURED' state
Validate Database Version Success Version '19.20.0.0.230718' for None
database 'MYBRCDB' is supported
Validate Database Datapatch Success Database 'MYBRCDB' is completely None
Application Status applied with datapatch
Validate TDE wallet presence Success Database 'MYBRCDB' is not TDE None
enabled. Skipping TDE wallet presence
check.
Validate Database Home Success Database home location check passed None
location for database MYBRCDB
Validate Database Status Success Database 'bsn01' is running and is in None
'CONFIGURED' state
Validate Database Version Success Version '19.19.0.0.230418' for None
database 'bsn01' is supported
Validate Database Datapatch Success Database 'bsn01' is completely None
Application Status applied with datapatch
Validate TDE wallet presence Success Database 'bsn01' is not TDE enabled. None
Skipping TDE wallet presence check.
Validate Database Home Success Database home location check passed None
location for database bsn01
Validate Database Status Success Database 'DBWEI' is running and is in None
'CONFIGURED' state
Validate Database Version Success Version '19.20.0.0.230718' for None
database 'DBWEI' is supported
Validate Database Datapatch Success Database 'DBWEI' is completely None
Application Status applied with datapatch
Validate TDE wallet presence Success Database 'DBWEI' is not TDE enabled. None
Skipping TDE wallet presence check.
Validate Database Home Success Database home location check passed None
location for database DBWEI
Validate Database Status Success Database 'LTTST' is running and is in None
'CONFIGURED' state
Validate Database Version Success Version '19.20.0.0.230718' for None
database 'LTTST' is supported
Validate Database Datapatch Success Database 'LTTST' is completely None
Application Status applied with datapatch
Validate TDE wallet presence Success Database 'LTTST' is not TDE enabled. None
Skipping TDE wallet presence check.
Validate Database Home Success Database home location check passed None
location for database LTTST
__CERTIFICATES__
Check using custom Success Using Default key pair None
certificates
Check the agent of the DB Success All the agents of the DB systems are None
System accessible accessible
__DBSYSTEMS__
Validate DB System DCS Failed testdbs02: FAIL Please ensure DCS components version
component versions [[dcscontroller]:19.20.0.0.0] match between DB System and BM
[[dcsagent]:19.20.0.0.0]
[[dcsadmin]:19.20.0.0.0]
Validate DB System DCS Failed testdbs01: FAIL Please ensure DCS components version
component versions [[dcscontroller]:19.20.0.0.0] match between DB System and BM
[[dcsagent]:19.20.0.0.0]
[[dcsadmin]:19.20.0.0.0]On my test system, multiple problems are detected.
First, I have an orphan database. As this ODA is a sandbox for multiple people, it’s not a problem removing the associated lines from the ODA registry (don’t do that on your production system):
systemctl stop initdcsagent
cd /opt/oracle/dcs/mysql/bin/
./mysql -u root --socket=/opt/oracle/dcs/mysql/log/mysqldb.sock
use dcsagentdb;
mysql> delete from ATTACHED_NETWORKS where db_id='e490fe71-d35e-404f-b799-f8fdb388f21f';
Query OK, 1 row affected (0.00 sec)
mysql> commit;
Query OK, 0 rows affected (0.00 sec)
mysql> delete from db where name='TST23';
Query OK, 1 row affected (0.00 sec)
mysql> commit;
Query OK, 0 rows affected (0.00 sec)
exit;
systemctl start initdcsagentI also need to upgrade the DCS components inside each DB System:
ssh [email protected]
odacli update-dcsadmin -v 19.21.0.0.0
sleep 30 ; odacli update-dcscomponents -v 19.21.0.0.0
odacli update-dcsagent -v 19.21.0.0.0
...
Now I can go back to my ODA and run the pre-upgrade report again:
odacli describe-preupgradereport -i 9526eb2a-6ffa-4f7f-b558-c5a10c09ebc2
Upgrade pre-check report
------------------------------------------------------------------------
Job ID: 9526eb2a-6ffa-4f7f-b558-c5a10c09ebc2
Description: Run pre-upgrade checks for Bare Metal
Status: SUCCESS
Created: January 9, 2024 5:04:55 PM CET
Result: All pre-checks succeeded
Node Name
---------------
dbi-oda1
Check Status Message Action
------------------------------ -------- -------------------------------------- --------------------------------------
__GI__
Check presence of databases Success No additional database found None
not managed by ODA registered in CRS
Check custom filesystems Success All file systems are owned and used None
by OS users provisioned by ODA
Check presence of HAVIP Success No HAVIP resources found registered None
resources not managed by ODA in CRS
Check presence of export Success No EXPORT resources found registered None
resources not managed by ODA in CRS
__OS__
Check Required OS files Success All the required files are present None
Check Additional OS RPMs Success No RPMs outside of base ISO were None
found on the system
__STORAGE__
Check Required Storage files Success All the required files are present None
Validate OAK Disks Success All OAK disks are in valid state None
Validate ASM Disk Groups Success All ASM disk groups are in valid state None
Validate ASM Disks Success All ASM disks are in valid state None
Check Database Home Storage Success The volume(s) orahome_sh,odabase_n0 None
volumes state is CONFIGURED.
Check space under /opt Success Free space on /opt: 10441.57 MB is None
more than required space: 1188 MB
Check space in ASM disk Success Space required for creating local None
group(s) homes is present in ACFS database
home storage. Required: 0 GB
Available: 38 GB
__SYS__
Validate Hardware Type Success Current hardware is supported None
Validate ILOM interconnect Success ILOM interconnect is not enabled None
Validate System Version Success System version 19.21.0.0.0 is None
supported
Verify System Timezone Success Succesfully verified the time zone None
file
Verify Grid User Success Grid user is verified None
Verify Grid Version Success Oracle Grid Infrastructure is running None
on the '19.20.0.0.230718' version on
all nodes
Check Audit Files Success Local Audit files not found None
__DB__
Validate Database Status Success Database 'DBITST' is running and is None
in 'CONFIGURED' state
Validate Database Version Success Version '19.20.0.0.230718' for None
database 'DBITST' is supported
Validate Database Datapatch Success Database 'DBITST' is completely None
Application Status applied with datapatch
Validate TDE wallet presence Success Database 'DBITST' is not TDE enabled. None
Skipping TDE wallet presence check.
Validate Database Home Success Database home location check passed None
location for database DBITST
Validate Database Status Success Database 'WS12C' is running and is in None
'CONFIGURED' state
Validate Database Version Success Version '19.20.0.0.230718' for None
database 'WS12C' is supported
Validate Database Datapatch Success Database 'WS12C' is completely None
Application Status applied with datapatch
Validate TDE wallet presence Success Database 'WS12C' is not TDE enabled. None
Skipping TDE wallet presence check.
Validate Database Home Success Database home location check passed None
location for database WS12C_SITE1
Validate Database Status Success Database 'MYBRCDB' is running and is None
in 'CONFIGURED' state
Validate Database Version Success Version '19.20.0.0.230718' for None
database 'MYBRCDB' is supported
Validate Database Datapatch Success Database 'MYBRCDB' is completely None
Application Status applied with datapatch
Validate TDE wallet presence Success Database 'MYBRCDB' is not TDE None
enabled. Skipping TDE wallet presence
check.
Validate Database Home Success Database home location check passed None
location for database MYBRCDB
Validate Database Status Success Database 'bsn01' is running and is in None
'CONFIGURED' state
Validate Database Version Success Version '19.19.0.0.230418' for None
database 'bsn01' is supported
Validate Database Datapatch Success Database 'bsn01' is completely None
Application Status applied with datapatch
Validate TDE wallet presence Success Database 'bsn01' is not TDE enabled. None
Skipping TDE wallet presence check.
Validate Database Home Success Database home location check passed None
location for database bsn01
Validate Database Status Success Database 'DBWEI' is running and is in None
'CONFIGURED' state
Validate Database Version Success Version '19.20.0.0.230718' for None
database 'DBWEI' is supported
Validate Database Datapatch Success Database 'DBWEI' is completely None
Application Status applied with datapatch
Validate TDE wallet presence Success Database 'DBWEI' is not TDE enabled. None
Skipping TDE wallet presence check.
Validate Database Home Success Database home location check passed None
location for database DBWEI
Validate Database Status Success Database 'LTTST' is running and is in None
'CONFIGURED' state
Validate Database Version Success Version '19.20.0.0.230718' for None
database 'LTTST' is supported
Validate Database Datapatch Success Database 'LTTST' is completely None
Application Status applied with datapatch
Validate TDE wallet presence Success Database 'LTTST' is not TDE enabled. None
Skipping TDE wallet presence check.
Validate Database Home Success Database home location check passed None
location for database LTTST
__CERTIFICATES__
Check using custom Success Using Default key pair None
certificates
Check the agent of the DB Success All the agents of the DB systems are None
System accessible accessible
__DBSYSTEMS__
Validate DB System DCS Success testdbs01: SUCCESS None
component versions
Validate DB System DCS Success testdbs02: SUCCESS None
component versionsDetach the node
Now, everything is fine and ready for unplugging my ODA configuration:
odacli detach-node -all
********************************************************************************
IMPORTANT
********************************************************************************
'odacli detach-node' will bring down the databases and grid services on the
system. The files that belong to the databases, which are stored on ASM or ACFS,
are left intact on the storage. The databases will be started up back after
re-imaging the ODA system using 'odacli restore-node' commands. As a good
precautionary measure, please backup all the databases on the system before you
start this process. Do not store the backup on this ODA machine since the local
file system will be wiped out as part of the re-image.
********************************************************************************
Do you want to continue (yes/no)[no] : yes
{
"jobId" : "5b613165-efbb-4de9-b737-2425e6b8471e",
"status" : "Created",
"message" : "On successful job completion, the server archive file will be generated at /opt/oracle/oak/restore/out. Please copy the server archive file outside of the ODA system before re-image",
"reports" : [ ],
"createTimestamp" : "January 09, 2024 17:09:31 PM CET",
"resourceList" : [ ],
"description" : "Detach node service creation for upgrade",
"updatedTime" : "January 09, 2024 17:09:31 PM CET",
"jobType" : null
}
odacli describe-job -i "5b613165-efbb-4de9-b737-2425e6b8471e"
Job details
----------------------------------------------------------------
ID: 5b613165-efbb-4de9-b737-2425e6b8471e
Description: Detach node service creation for upgrade
Status: Success
Created: January 9, 2024 5:09:31 PM CET
Message: On successful job completion, the server archive file will be generated at /opt/oracle/oak/restore/out. Please copy the server archive file outside of the ODA system before re-image
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------
Creating initialization file January 9, 2024 5:09:32 PM CET January 9, 2024 5:09:32 PM CET Success
Creating firstnet response file January 9, 2024 5:09:32 PM CET January 9, 2024 5:09:32 PM CET Success
Saving system information January 9, 2024 5:09:32 PM CET January 9, 2024 5:09:32 PM CET Success
Saving Appliance data January 9, 2024 5:09:32 PM CET January 9, 2024 5:11:26 PM CET Success
Saving OS files January 9, 2024 5:09:32 PM CET January 9, 2024 5:09:32 PM CET Success
Saving CPU cores information January 9, 2024 5:09:32 PM CET January 9, 2024 5:09:32 PM CET Success
Saving storage files January 9, 2024 5:09:32 PM CET January 9, 2024 5:09:32 PM CET Success
Saving Volumes January 9, 2024 5:09:32 PM CET January 9, 2024 5:10:02 PM CET Success
Saving File Systems January 9, 2024 5:10:02 PM CET January 9, 2024 5:10:26 PM CET Success
Saving Quorum Disks January 9, 2024 5:10:26 PM CET January 9, 2024 5:10:27 PM CET Success
Saving System January 9, 2024 5:10:27 PM CET January 9, 2024 5:10:28 PM CET Success
Saving Networks January 9, 2024 5:10:28 PM CET January 9, 2024 5:10:28 PM CET Success
Saving Database Storages January 9, 2024 5:10:28 PM CET January 9, 2024 5:10:46 PM CET Success
Saving Database Homes January 9, 2024 5:10:46 PM CET January 9, 2024 5:10:46 PM CET Success
+-- Saving OraDB19000_home2 January 9, 2024 5:10:46 PM CET January 9, 2024 5:10:46 PM CET Success
+-- Saving OraDB19000_home3 January 9, 2024 5:10:46 PM CET January 9, 2024 5:10:46 PM CET Success
+-- Saving OraDB19000_home4 January 9, 2024 5:10:46 PM CET January 9, 2024 5:10:46 PM CET Success
+-- Saving OraDB19000_home6 January 9, 2024 5:10:46 PM CET January 9, 2024 5:10:46 PM CET Success
+-- Saving OraDB12201_home1 January 9, 2024 5:10:46 PM CET January 9, 2024 5:10:46 PM CET Success
Saving Databases January 9, 2024 5:10:46 PM CET January 9, 2024 5:11:12 PM CET Success
+-- Saving DBITST January 9, 2024 5:10:46 PM CET January 9, 2024 5:10:51 PM CET Success
+-- Saving WS12C_SITE1 January 9, 2024 5:10:51 PM CET January 9, 2024 5:10:54 PM CET Success
+-- Saving MYBRCDB January 9, 2024 5:10:54 PM CET January 9, 2024 5:10:59 PM CET Success
+-- Saving bsn01 January 9, 2024 5:10:59 PM CET January 9, 2024 5:11:04 PM CET Success
+-- Saving DBWEI January 9, 2024 5:11:04 PM CET January 9, 2024 5:11:09 PM CET Success
+-- Saving LTTST January 9, 2024 5:11:09 PM CET January 9, 2024 5:11:12 PM CET Success
Saving Object swift stores January 9, 2024 5:11:12 PM CET January 9, 2024 5:11:12 PM CET Success
Saving Database Backups January 9, 2024 5:11:12 PM CET January 9, 2024 5:11:12 PM CET Success
Saving NFS Backups January 9, 2024 5:11:12 PM CET January 9, 2024 5:11:12 PM CET Success
Saving Certificates and wallets Backup January 9, 2024 5:11:12 PM CET January 9, 2024 5:11:12 PM CET Success
Saving KVM black-box resources January 9, 2024 5:11:12 PM CET January 9, 2024 5:11:16 PM CET Success
Saving DB System Resources January 9, 2024 5:11:16 PM CET January 9, 2024 5:11:26 PM CET Success
Creating databases version list January 9, 2024 5:11:26 PM CET January 9, 2024 5:11:26 PM CET Success
Deconfiguring Appliance January 9, 2024 5:11:26 PM CET January 9, 2024 5:21:14 PM CET Success
Deconfiguring Databases January 9, 2024 5:11:26 PM CET January 9, 2024 5:14:17 PM CET Success
+-- Deconfiguring DBITST January 9, 2024 5:11:26 PM CET January 9, 2024 5:11:56 PM CET Success
+-- Deconfiguring WS12C_SITE1 January 9, 2024 5:11:56 PM CET January 9, 2024 5:12:21 PM CET Success
+-- Deconfiguring MYBRCDB January 9, 2024 5:12:21 PM CET January 9, 2024 5:12:48 PM CET Success
+-- Deconfiguring bsn01 January 9, 2024 5:12:48 PM CET January 9, 2024 5:13:20 PM CET Success
+-- Deconfiguring DBWEI January 9, 2024 5:13:20 PM CET January 9, 2024 5:13:57 PM CET Success
+-- Deconfiguring LTTST January 9, 2024 5:13:57 PM CET January 9, 2024 5:14:17 PM CET Success
Saving database backup reports January 9, 2024 5:14:17 PM CET January 9, 2024 5:14:17 PM CET Success
Resizing Quorum Disks January 9, 2024 5:14:17 PM CET January 9, 2024 5:14:17 PM CET Success
Deconfiguring Grid Infrastructure January 9, 2024 5:14:17 PM CET January 9, 2024 5:21:13 PM CET Success
Backup Quorum Disks January 9, 2024 5:21:13 PM CET January 9, 2024 5:21:14 PM CET Success
Creating the server archive files January 9, 2024 5:21:14 PM CET January 9, 2024 5:21:15 PM CET Success
Zipping the server archive files January 9, 2024 5:21:15 PM CET January 9, 2024 5:21:16 PM CET Success
VERY IMPORTANT: take a copy of files inside /opt/oracle/oak/restore/out OUTSIDE the ODA. I would recommend 2 copies on 2 different disks/systems. Check if you can open the zipfile on your computer. The zipfile name is normaly serverarchive_{hostname}.zip.
Reimage the ODA
WARNING: classic reimaging implies using the cleanup.pl script. You must not cleanup anything here: DATA disks content is needed for DPR!
Reimaging is done through the ILOM console as if your ODA were brand new. Connect the 19.21 ISO image, select CDROM as next boot drive and do a power cycle of the server. The OS will then be reinstalled from scratch and it should take about 1 hour.
Once your ODA is reimaged, don’t forget that credentials have been reset to root/welcome1.
Connect to the ODA remote console from ILOM interface.
Configure the initial network settings. You don’t need to remember these settings, you can find them in the configure-firstnet.rsp from the zipfile you picked up after the ODA detach-node operation:
cat configure-firstnet.rsp
HOSTNAME=dbi-oda1
INTERFACE_NAME=btbond1
VLAN=NO
IP_ADDR=10.36.0.241
SUBNET_MASK=255.255.255.0
GATEWAY=10.36.0.1Patch the microcodes
You now need to copy again the patch file to this “new” ODA. This is because reimaging don’t update the firmware/BIOS/ILOM/storage components:
mkdir /opt/dbi
-- patch copy hereLet’s unzip the patch again and then register it in the ODA registry:
unzip p35938481_1921000_Linux-x86-64.zip
odacli update-repository -f /opt/dbi/oda-sm-19.21.0.0.0-231220-server.zip
sleep 30 ; odacli describe-job -i "ed4fe6a7-12b4-47c5-a976-0097325d083a"
Job details
----------------------------------------------------------------
ID: ed4fe6a7-12b4-47c5-a976-0097325d083a
Description: Repository Update
Status: Success
Created: January 10, 2024 7:55:41 AM UTC
Message: /opt/dbi/oda-sm-19.21.0.0.0-231220-server.zip
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------
Unzip bundle January 10, 2024 7:55:41 AM UTC January 10, 2024 7:56:06 AM UTC Success
And now, a prepatch report is needed:
odacli create-prepatchreport -s -v 19.21.0.0.0
odacli describe-prepatchreport -i 5512952b-774e-4fab-a3b0-76b5baba1d5b
Patch pre-check report
------------------------------------------------------------------------
Job ID: 5512952b-774e-4fab-a3b0-76b5baba1d5b
Description: Patch pre-checks for [OS, ILOM, SERVER]
Status: SUCCESS
Created: January 10, 2024 7:56:56 AM UTC
Result: All pre-checks succeeded
Node Name
---------------
oak
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.21.0.0.0.
Is patch location available Success Patch location is available.
Verify OS patch Success There are no packages available for
an update
Validate command execution Success Skipped command execution verfication
- Instance is not provisioned
__ILOM__
Validate ILOM server reachable Success Successfully connected with ILOM
server using public IP and USB
interconnect
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.21.0.0.0.
Is patch location available Success Patch location is available.
Checking Ilom patch Version Success Successfully verified the versions
Patch location validation Success Successfully validated location
Validate command execution Success Skipped command execution verfication
- Instance is not provisioned
__SERVER__
Validate local patching Success Successfully validated server local
patching
Validate command execution Success Skipped command execution verfication
- Instance is not provisionedLet’s run the update-server. It will skip software updates as our system is brand new:
odacli update-server -v 19.21.0.0.0
odacli describe-job -i "ab669776-c7dd-4f8b-9453-e8c55ad465fa"
Job details
----------------------------------------------------------------
ID: ab669776-c7dd-4f8b-9453-e8c55ad465fa
Description: Server Patching
Status: Success
Created: January 10, 2024 8:09:08 AM UTC
Message:
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------
Validate ILOM server reachable January 10, 2024 8:09:09 AM UTC January 10, 2024 8:09:09 AM UTC Success
Stop DCS Admin January 10, 2024 8:09:10 AM UTC January 10, 2024 8:09:11 AM UTC Success
Generate mTLS certificates January 10, 2024 8:09:11 AM UTC January 10, 2024 8:09:12 AM UTC Success
Exporting Public Keys January 10, 2024 8:09:12 AM UTC January 10, 2024 8:09:14 AM UTC Success
Creating Trust Store January 10, 2024 8:09:14 AM UTC January 10, 2024 8:09:17 AM UTC Success
Update config files January 10, 2024 8:09:17 AM UTC January 10, 2024 8:09:17 AM UTC Success
Restart DCS Admin January 10, 2024 8:09:17 AM UTC January 10, 2024 8:09:38 AM UTC Success
Modify BM udev rules January 10, 2024 8:09:38 AM UTC January 10, 2024 8:09:58 AM UTC Success
Creating repositories using yum January 10, 2024 8:09:58 AM UTC January 10, 2024 8:10:01 AM UTC Success
Updating YumPluginVersionLock rpm January 10, 2024 8:10:01 AM UTC January 10, 2024 8:10:01 AM UTC Success
Applying OS Patches January 10, 2024 8:10:02 AM UTC January 10, 2024 8:10:06 AM UTC Success
Creating repositories using yum January 10, 2024 8:10:06 AM UTC January 10, 2024 8:10:06 AM UTC Success
Applying HMP Patches January 10, 2024 8:10:06 AM UTC January 10, 2024 8:10:09 AM UTC Success
Patch location validation January 10, 2024 8:10:09 AM UTC January 10, 2024 8:10:09 AM UTC Success
Oda-hw-mgmt upgrade January 10, 2024 8:10:09 AM UTC January 10, 2024 8:10:09 AM UTC Success
Applying Firmware Disk Patches January 10, 2024 8:10:10 AM UTC January 10, 2024 8:10:13 AM UTC Success
Applying Firmware Controller Patches January 10, 2024 8:10:13 AM UTC January 10, 2024 8:15:07 AM UTC Success
Checking Ilom patch Version January 10, 2024 8:15:07 AM UTC January 10, 2024 8:15:07 AM UTC Success
Patch location validation January 10, 2024 8:15:07 AM UTC January 10, 2024 8:15:07 AM UTC Success
Save password in Wallet January 10, 2024 8:15:07 AM UTC January 10, 2024 8:15:08 AM UTC Success
Disabling IPMI v2 January 10, 2024 8:15:08 AM UTC January 10, 2024 8:15:08 AM UTC Success
Apply Ilom patch January 10, 2024 8:15:08 AM UTC January 10, 2024 8:25:12 AM UTC Success
Copying Flash Bios to Temp location January 10, 2024 8:25:12 AM UTC January 10, 2024 8:25:12 AM UTC Success
Update System version January 10, 2024 8:25:13 AM UTC January 10, 2024 8:25:13 AM UTC Success
Cleanup JRE Home January 10, 2024 8:25:13 AM UTC January 10, 2024 8:25:13 AM UTC Success
Generating and saving BOM January 10, 2024 8:25:13 AM UTC January 10, 2024 8:25:17 AM UTC Success
PreRebootNode Actions January 10, 2024 8:25:17 AM UTC January 10, 2024 8:25:17 AM UTC Success
Reboot Ilom January 10, 2024 8:25:17 AM UTC January 10, 2024 8:25:17 AM UTC Success
It was not needed on my ODA coming from 19.20, but you may also need to update your storage:
odacli update-storage -v 19.21.0.0.0
...
odacli describe-component
System Version
---------------
19.21.0.0.0
System node Name
---------------
oak
Local System Version
---------------
19.21.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK
19.21.0.0.0 up-to-date
DCSCONTROLLER
19.21.0.0.0 up-to-date
DCSCLI
19.21.0.0.0 up-to-date
DCSAGENT
19.21.0.0.0 up-to-date
DCSADMIN
19.21.0.0.0 up-to-date
OS
8.8 up-to-date
ILOM
5.1.3.20.r153596 up-to-date
BIOS
52110200 up-to-date
LOCAL CONTROLLER FIRMWARE
8000D9AB up-to-date
SHARED CONTROLLER FIRMWARE
VDV1RL06 up-to-date
LOCAL DISK FIRMWARE
XC311132 up-to-date
HMP
2.4.9.1.601 up-to-date
Register the GI clone and the “unplug” file
Copy the 19.21 GI clone and zipfile resulting from detach-node operation:
unzip p30403673_1921000_Linux-x86-64.zip
odacli update-repository -f /opt/dbi/odacli-dcs-19.21.0.0.0-231211-GI-19.21.0.0.zip
sleep 30 ; odacli describe-job -i "d89b9524-792f-4b14-9b84-e76fd048cafa"
Job details
----------------------------------------------------------------
ID: d89b9524-792f-4b14-9b84-e76fd048cafa
Description: Repository Update
Status: Success
Created: January 10, 2024 8:44:19 AM UTC
Message: /opt/dbi/odacli-dcs-19.21.0.0.0-231211-GI-19.21.0.0.zip
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------
Unzip bundle January 10, 2024 8:44:20 AM UTC January 10, 2024 8:44:51 AM UTC Success
odacli update-repository -f /opt/dbi/serverarchive_dbi-oda1.zip
sleep 10 ; odacli describe-job -i "4ddd1a16-3c5e-4f05-a81b-00d16032f53c"
Job details
----------------------------------------------------------------
ID: 4ddd1a16-3c5e-4f05-a81b-00d16032f53c
Description: Repository Update
Status: Success
Created: January 10, 2024 8:46:42 AM UTC
Message: /opt/dbi/serverarchive_dbi-oda1.zip
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------
Unzip bundle January 10, 2024 8:46:42 AM UTC January 10, 2024 8:46:43 AM UTC SuccessRestore the GI configuration
Now it’s time to restore your GI configuration on top of a new GI home setup, it’s done with the restore-node operation:
odacli restore-node -g
Enter New system password:
Retype New system password:
Enter an initial password for Web Console account (oda-admin):
Retype the password for Web Console account (oda-admin):
User 'oda-admin' created successfully...
DCS-10001:Internal error encountered: Failed to validate provision request /opt/oracle/oak/restore/metadata/provisionInstance.json: DCS-10045:Validation error encountered: Hostname 'dbi-oda1' does not resolve to a unique IP address..
At this time, your ODA may not be able to use DNS resolution for host and ILOM hostnames, you can add them to your /etc/hosts file and retry the restore-node:
vi /etc/hosts
10.36.0.241 dbi-oda1
10.36.0.240 dbi-oda01-ilom
odacli restore-node -g
Enter New system password:
Retype New system password:
odacli describe-job -i "6183aa14-fdb4-440a-be0c-c3d673082df4"
Job details
----------------------------------------------------------------
ID: 6183aa14-fdb4-440a-be0c-c3d673082df4
Description: Restore node service - GI
Status: Failure (To view Error Correlation report, run "odacli describe-job -i 6183aa14-fdb4-440a-be0c-c3d673082df4 --ecr" command)
Created: January 10, 2024 9:54:34 AM CET
Message: DCS-10001:Internal error encountered: Failed to restart network service.
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------
Restore node service creation January 10, 2024 9:54:46 AM CET January 10, 2024 9:55:03 AM CET Failure
Restore node service creation January 10, 2024 9:54:47 AM CET January 10, 2024 9:55:03 AM CET Failure
Restore node service creation January 10, 2024 9:54:47 AM CET January 10, 2024 9:55:03 AM CET Failure
Provisioning service creation January 10, 2024 9:54:47 AM CET January 10, 2024 9:55:03 AM CET Failure
Provisioning service creation January 10, 2024 9:54:47 AM CET January 10, 2024 9:55:03 AM CET Failure
Network setup January 10, 2024 9:54:49 AM CET January 10, 2024 9:55:03 AM CET Failure
Provisioning service creation January 10, 2024 9:54:50 AM CET January 10, 2024 9:55:03 AM CET Failure
Setting up Network January 10, 2024 9:54:51 AM CET January 10, 2024 9:54:51 AM CET Success
Provisioning service creation January 10, 2024 9:54:51 AM CET January 10, 2024 9:55:03 AM CET Failure
Restarting Network Service January 10, 2024 9:54:51 AM CET January 10, 2024 9:55:03 AM CET FailureUnfortunately, it didn’t work for me at first try, let’s do the troubleshooting:
odacli describe-job -i 6183aa14-fdb4-440a-be0c-c3d673082df4 --ecr
ODA Assistant - Error Correlation report
----------------------------------------
Failed job ID: 6183aa14-fdb4-440a-be0c-c3d673082df4
Description: Restore node service - GI
Start Time: 2024-01-10 08:54:46
End Time: 2024-01-10 08:55:03
EC report path: /opt/oracle/dcs/da/da_repo/d051b53d-e35f-4374-8c6f-7e840b144b63.json
Failed Task Messages
--------------------
[Restore node service - GI] - DCS-10001:Internal error encountered: Failed to restart network service.
dbi-oda1 Log Messages
----------------------------
DCS Agent
~~~~~~~~~
Error Logs
==========
[Restarting Network Service] - Failed to restart service 'network'
[Restarting Network Service] - Fail to restart network service
[Restarting Network Service] - Task got exception:
[Provisioning service creation] - DCS-10001:Internal error encountered: Failed to restart network service.
Error code - DCS-10001
Cause: An internal error occurred.
Action: Contact Oracle Support for assistance.
Warning Logs
============
[[ SEND-THREAD 124 ]] - [ [ SEND-THREAD 124 ] dcs0-priv:22001] Request failed: Operation: GET Host: dcs0-priv:22001 Path: /joblocks/6183aa14-fdb4-440a-be0c-c3d673082df4 Data: null Status: 404
Release Notes
-------------
No matching results were found.
Documentation
-------------
1. Error in restore node process in Data Preserving Reprovisioning
Abstract - In the Data Preserving Reprovisioning process, during node restore, an error may be encountered.
Link - https://dbi-oda1.dbi-lab.ch:7093/docs/cmtrn/issues-with-oda-odacli.html#GUID-F1385628-9F87-4FEF-8D27-289A3ED459EC
2. Error in restore node process in Data Preserving Reprovisioning
Abstract - In the Data Preserving Reprovisioning process, during node restore, an error may be encountered.
Link - https://dbi-oda1.dbi-lab.ch:7093/docs/cmtrn/issues-with-oda-odacli.html#GUID-75D52887-D425-4753-AF44-EFAB5C148873
3. Managing Backup, Restore, and Recovery on a Database System in a KVM Deployment
Abstract - Understand the backup, restore, and recovery operations supported on a database system in a KVM deployment.
Link - https://dbi-oda1.dbi-lab.ch:7093/docs/cmtxn/managing-oracle-database-appliance-kvm-deployment1.html#GUID-7318F4D7-4CB8-486C-9DC7-A7490925B866
4. Backup, Restore and Recover Databases
Abstract - Review backup, restore, and recovery operations for your databases.
Link - https://dbi-oda1.dbi-lab.ch:7093/docs/cmtxn/backup-recover-restore.html#GUID-032C43EC-20B9-4036-ADA9-7631EEBBFEF6
5. Reimaging an Oracle Database Appliance Baremetal System
Abstract - Reimage Oracle Database Appliance to perform a bare metal restore of Oracle Database Appliance.
Link - https://dbi-oda1.dbi-lab.ch:7093/docs/cmtxn/reimaging-oda.html#GUID-C4D52E9A-C516-4B4F-829F-E102317AE986
NOTE: For additional details such as file name and line numbers of error logs, please refer to /opt/oracle/dcs/da/da_repo/d051b53d-e35f-4374-8c6f-7e840b144b63.jsonSomething is wrong with my network, let’s try to restart it:
service network restart
Restarting network (via systemctl): Job for network.service failed because the control process exited with error code.
See "systemctl status network.service" and "journalctl -xe" for details.
[FAILED]
systemctl status network.service
● network.service - LSB: Bring up/down networking
Loaded: loaded (/etc/rc.d/init.d/network; generated)
Active: failed (Result: exit-code) since Wed 2024-01-10 11:41:13 CET; 3s ago
Docs: man:systemd-sysv-generator(8)
Process: 6948 ExecStart=/etc/rc.d/init.d/network start (code=exited, status=1/FAILURE)
Tasks: 0 (limit: 79998)
Memory: 4.2M
CGroup: /system.slice/network.service
Jan 10 11:41:13 dbi-oda1 network[7709]: RTNETLINK answers: File exists
Jan 10 11:41:13 dbi-oda1 network[7710]: RTNETLINK answers: File exists
Jan 10 11:41:13 dbi-oda1 network[7711]: RTNETLINK answers: File exists
Jan 10 11:41:13 dbi-oda1 network[7712]: RTNETLINK answers: File exists
Jan 10 11:41:13 dbi-oda1 network[7713]: RTNETLINK answers: File exists
Jan 10 11:41:13 dbi-oda1 network[7714]: RTNETLINK answers: File exists
Jan 10 11:41:13 dbi-oda1 network[7715]: RTNETLINK answers: File exists
Jan 10 11:41:13 dbi-oda1 systemd[1]: network.service: Control process exited, code=exited status=1
Jan 10 11:41:13 dbi-oda1 systemd[1]: network.service: Failed with result 'exit-code'.
Jan 10 11:41:13 dbi-oda1 systemd[1]: Failed to start LSB: Bring up/down networking.
tail /var/log/messages
Jan 10 11:47:47 dbi-oda1 network[8003]: Bringing up interface enp0s20f0u8u3c2:
Jan 10 11:47:47 dbi-oda1 network[8534]: Error: Connection activation failed: No suitable device found for this connection (device virbr0 not available because profile is not compatible with device (mismatching interface name)).
Jan 10 11:47:47 dbi-oda1 network[8003]: [FAILED]
ls -lrt /etc/sysconfig/network-scripts/ifcfg-*
-rw-r--r--. 1 root root 160 Jan 9 18:54 /etc/sysconfig/network-scripts/ifcfg-em1
-rw-r--r--. 1 root root 186 Jan 9 18:54 /etc/sysconfig/network-scripts/ifcfg-p7p3
-rw-r--r--. 1 root root 186 Jan 9 18:54 /etc/sysconfig/network-scripts/ifcfg-p7p4
-rw-r--r--. 1 root root 218 Jan 9 18:54 /etc/sysconfig/network-scripts/ifcfg-btbond2
-rw-r--r--. 1 root root 186 Jan 9 18:54 /etc/sysconfig/network-scripts/ifcfg-p7p1
-rw-r--r--. 1 root root 186 Jan 9 18:54 /etc/sysconfig/network-scripts/ifcfg-p7p2
-rw-r--r--. 1 root root 264 Jan 9 18:54 /etc/sysconfig/network-scripts/ifcfg-lo
-rw-r--r--. 1 root root 177 Jan 9 18:55 /etc/sysconfig/network-scripts/ifcfg-priv0
-rw-r--r-- 1 root root 263 Jan 10 08:26 /etc/sysconfig/network-scripts/ifcfg-btbond1
-rw-r--r-- 1 root root 136 Jan 10 09:54 /etc/sysconfig/network-scripts/ifcfg-pubnet
-rw-r--r-- 1 root root 146 Jan 10 11:48 /etc/sysconfig/network-scripts/ifcfg-enp0s20f0u8u3c2
I don’t know where interface enp0s20f0u8u3c2 come from. My ODA is using a single Ethernet card with 4 ports (p7px), the em1 interface is there but never used on ODA, the 2-port bondings are here, pubnet virtual interface is here. But enp0s20f0u8u3c2 shouldn’t be there. Let’s remove the script and restart the network:
mv /etc/sysconfig/network-scripts/ifcfg-enp0s20f0u8u3c2 /opt/dbi/
service network restart
Restarting network (via systemctl): [ OK ]It’s better now.
Let’s retry the restore-node:
odacli restore-node -g
Enter New system password:
Retype New system password:
DCS-10045:Validation error encountered: System has already undergone provisioning with status 'FAILED'.
OK, doing the restore-node is no more possible because it already restored some stuff. I need to do a cleanup, and the cleanup MUST destect that DPR is used:
/opt/oracle/oak/onecmd/cleanup.pl
INFO: Log file is /opt/oracle/oak/log/dbi-oda1/cleanup/cleanup_2024-01-23_10-36-34.log
INFO: Log file is /opt/oracle/oak/log/dbi-oda1/cleanup/dcsemu_diag_precleanup_2024-01-23_10-36-34.log
INFO: Platform is 'BM'
INFO: *******************************************************************
INFO: ** Starting process to cleanup provisioned host dbi-oda1 **
INFO: *******************************************************************
WARNING: DPR environment detected. DPR specific cleanup involves
WARNING: deconfiguring the ODA software stack without touching ASM
WARNING: storage to allow rerunning of the 'odacli restore-node -g'
WARNING: command. If regular cleanup(which erases ASM disk headers)
WARNING: is intended, rerun cleanup.pl with '-nodpr' option.
WARNING: If Multi-User Access is enabled, use '-omausers' option to
WARNING: delete the custom users created during the previous run.
Do you want to continue (yes/no) : yes
INFO:
Running cleanup will delete Grid User - 'grid' and
INFO: DB user - 'oracle' and also the
INFO: groups 'oinstall,dba,asmadmin,asmoper,asmdba'
INFO: nodes will be rebooted
Do you want to continue (yes/no) : yes
INFO: *************************************
INFO: ** Checking for GI bits presence
INFO: *************************************
INFO: No Grid Infrastructure files were found on the system in the /u01/app directory.
INFO: *************************************
INFO: ** DPR Cleanup
INFO: *************************************
INFO: Nothing to do.
SUCCESS: DPR cleanup actions completed.
INFO: Attempting to stop DCS agent on local node
INFO: *************************************
INFO: ** Executing AFD cleanup commands
INFO: *************************************
INFO: *************************************
INFO: ** Cleaning Oracle HAMI for ODA
INFO: *************************************
INFO: ** - Oracle HAMI for ODA - ensembles cleaned successfully
INFO: ** - Oracle HAMI for ODA - users cleaned successfully
INFO: *************************************
INFO: ** Executing stack deinstall commands
INFO: *************************************
INFO: *************************************
INFO: ** Removing IPC objects
INFO: *************************************
pgrep: invalid user name: oracle
INFO: Cleaning up IDM configurations...
Deleting directory </opt/oracle/dcs/idm>INFO: *************************************
INFO: ** Cleaning miscellaneous components:
INFO: *************************************
INFO: ** - reset limits.conf
INFO: ** - delete users
INFO: ** - delete groups
INFO: ** - hostname, gateway and hosts reset commands
INFO: ** - dcs cleanup and orphan files removal commands
INFO: Attempting to clean MySQL tables on local node
INFO: Cleaning up network bridges
INFO: default net is: pubnet
INFO: /etc/sysconfig/network-scripts/ifcfg-btbond1 has an ip-address stored in it
INFO: *************************************
INFO: ** Removing KVM files
INFO: *************************************
INFO: *************************************
INFO: ** Removing BM CPU Pool files
INFO: *************************************
INFO: ** - networking cleaning commands
INFO: ** - UTC reset commands
INFO: *************************************
INFO: ** Removing Oracle AHF RPM
INFO: *************************************
INFO: No need to uninstall Oracle AHF RPM
INFO: Cleaning up ACFS mounts...
INFO: Reset password for 'root' to default value
INFO: Executing <command to reset root password to default value>
INFO: Removing SSH keys on dbi-oda1
INFO: Rebooting the system via <reboot>...
INFO: Executing <reboot>
INFO: Cleanup was successful
INFO: Log file is /opt/oracle/oak/log/dbi-oda1/cleanup/cleanup_2024-01-23_10-36-34.log
WARNING: After system reboot, please re-run "odacli update-repository" for GI/DB clones,
WARNING: before running "odacli restore-node -g".Now my system reboots and I will need to redo the configure-firstnet for inital network configuration. Hopefully, cleanup will keep my /opt/dbi folder with all the needed files. I just need to register the files into the ODA registry again:
odacli update-repository -f /opt/dbi/oda-sm-19.21.0.0.0-231220-server.zip
sleep 60 ; odacli describe-job -i "8a70333a-ebfd-419f-8e99-09b150bf627e"
Job details
----------------------------------------------------------------
ID: 8a70333a-ebfd-419f-8e99-09b150bf627e
Description: Repository Update
Status: Success
Created: January 23, 2024 9:59:18 AM UTC
Message: /opt/dbi/oda-sm-19.21.0.0.0-231220-server.zip
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------
Unzip bundle January 23, 2024 9:59:18 AM UTC January 23, 2024 9:59:43 AM UTC Success
odacli create-prepatchreport -s -v 19.21.0.0.0
sleep 120 ; odacli describe-prepatchreport -i 071ece64-a9be-4f11-a3c0-8aec8c39cb14
Patch pre-check report
------------------------------------------------------------------------
Job ID: 071ece64-a9be-4f11-a3c0-8aec8c39cb14
Description: Patch pre-checks for [OS, ILOM, SERVER]
Status: SUCCESS
Created: January 23, 2024 10:02:05 AM UTC
Result: All pre-checks succeeded
Node Name
---------------
oak
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.21.0.0.0.
Is patch location available Success Patch location is available.
Verify OS patch Success There are no packages available for
an update
Validate command execution Success Skipped command execution verfication
- Instance is not provisioned
__ILOM__
Validate ILOM server reachable Success Successfully connected with ILOM
server using public IP and USB
interconnect
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.21.0.0.0.
Is patch location available Success Patch location is available.
Checking Ilom patch Version Success Successfully verified the versions
Patch location validation Success Successfully validated location
Validate command execution Success Skipped command execution verfication
- Instance is not provisioned
__SERVER__
Validate local patching Success Successfully validated server local
patching
Validate command execution Success Skipped command execution verfication
- Instance is not provisioned
odacli update-repository -f /opt/dbi/serverarchive_dbi-oda1.zip
sleep 10 ; odacli describe-job -i "4f1b792c-6c12-4e15-8d4f-439526df91b4"
Job details
----------------------------------------------------------------
ID: 4f1b792c-6c12-4e15-8d4f-439526df91b4
Description: Repository Update
Status: Success
Created: January 23, 2024 10:07:22 AM UTC
Message: /opt/dbi/serverarchive_dbi-oda1.zip
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------
Unzip bundle January 23, 2024 10:07:22 AM UTC January 23, 2024 10:07:22 AM UTC Success
echo "10.36.0.241 dbi-oda1" >> /etc/hosts
echo "10.36.0.240 dbi-oda01-ilom" >> /etc/hosts
odacli restore-node -g
odacli describe-job -i c8b64bc7-5a54-4e77-b8f7-0bd3368c2ec0
Job details
----------------------------------------------------------------
ID: c8b64bc7-5a54-4e77-b8f7-0bd3368c2ec0
Description: Restore node service - GI
Status: Success
Created: January 23, 2024 9:11:49 AM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------
Restore node service creation January 23, 2024 9:11:57 AM CET January 23, 2024 9:39:19 AM CET Success
Setting up Network January 23, 2024 9:11:59 AM CET January 23, 2024 9:11:59 AM CET Success
Setting up Vlan January 23, 2024 9:12:27 AM CET January 23, 2024 9:12:29 AM CET Success
Setting up Network January 23, 2024 9:13:03 AM CET January 23, 2024 9:13:03 AM CET Success
Network update January 23, 2024 9:13:39 AM CET January 23, 2024 9:14:10 AM CET Success
Updating network January 23, 2024 9:13:39 AM CET January 23, 2024 9:14:10 AM CET Success
Setting up Network January 23, 2024 9:13:39 AM CET January 23, 2024 9:13:39 AM CET Success
OS usergroup 'asmdba' creation January 23, 2024 9:14:10 AM CET January 23, 2024 9:14:10 AM CET Success
OS usergroup 'asmoper' creation January 23, 2024 9:14:10 AM CET January 23, 2024 9:14:10 AM CET Success
OS usergroup 'asmadmin' creation January 23, 2024 9:14:10 AM CET January 23, 2024 9:14:10 AM CET Success
OS usergroup 'dba' creation January 23, 2024 9:14:10 AM CET January 23, 2024 9:14:10 AM CET Success
OS usergroup 'dbaoper' creation January 23, 2024 9:14:10 AM CET January 23, 2024 9:14:11 AM CET Success
OS usergroup 'oinstall' creation January 23, 2024 9:14:11 AM CET January 23, 2024 9:14:11 AM CET Success
OS user 'grid' creation January 23, 2024 9:14:11 AM CET January 23, 2024 9:14:11 AM CET Success
OS user 'oracle' creation January 23, 2024 9:14:11 AM CET January 23, 2024 9:14:11 AM CET Success
Default backup policy creation January 23, 2024 9:14:11 AM CET January 23, 2024 9:14:12 AM CET Success
Backup Config name validation January 23, 2024 9:14:12 AM CET January 23, 2024 9:14:12 AM CET Success
Backup config metadata persist January 23, 2024 9:14:12 AM CET January 23, 2024 9:14:12 AM CET Success
Grant permission to RHP files January 23, 2024 9:14:12 AM CET January 23, 2024 9:14:12 AM CET Success
Add SYSNAME in Env January 23, 2024 9:14:12 AM CET January 23, 2024 9:14:12 AM CET Success
Install oracle-ahf January 23, 2024 9:14:12 AM CET January 23, 2024 9:17:04 AM CET Success
Stop DCS Admin January 23, 2024 9:17:41 AM CET January 23, 2024 9:17:41 AM CET Success
Generate mTLS certificates January 23, 2024 9:17:41 AM CET January 23, 2024 9:17:43 AM CET Success
Exporting Public Keys January 23, 2024 9:17:43 AM CET January 23, 2024 9:17:45 AM CET Success
Creating Trust Store January 23, 2024 9:17:45 AM CET January 23, 2024 9:17:49 AM CET Success
Update config files January 23, 2024 9:17:49 AM CET January 23, 2024 9:17:49 AM CET Success
Restart DCS Admin January 23, 2024 9:17:49 AM CET January 23, 2024 9:18:09 AM CET Success
Unzipping storage configuration files January 23, 2024 9:18:09 AM CET January 23, 2024 9:18:09 AM CET Success
Reloading multipath devices January 23, 2024 9:18:10 AM CET January 23, 2024 9:18:10 AM CET Success
Restart oakd January 23, 2024 9:18:10 AM CET January 23, 2024 9:18:21 AM CET Success
Restore Quorum Disks January 23, 2024 9:18:48 AM CET January 23, 2024 9:18:49 AM CET Success
Creating GI home directories January 23, 2024 9:18:49 AM CET January 23, 2024 9:18:49 AM CET Success
Extract GI clone January 23, 2024 9:18:49 AM CET January 23, 2024 9:19:54 AM CET Success
Restart oakd January 23, 2024 9:19:31 AM CET January 23, 2024 9:18:48 AM CET Success
Creating wallet for Root User January 23, 2024 9:19:54 AM CET January 23, 2024 9:19:59 AM CET Success
Creating wallet for ASM Client January 23, 2024 9:19:59 AM CET January 23, 2024 9:20:03 AM CET Success
Grid stack creation January 23, 2024 9:20:03 AM CET January 23, 2024 9:30:47 AM CET Success
GI Restore with RHP January 23, 2024 9:20:03 AM CET January 23, 2024 9:27:32 AM CET Success
Updating GIHome version January 23, 2024 9:27:33 AM CET January 23, 2024 9:27:37 AM CET Success
Post cluster OAKD configuration January 23, 2024 9:30:47 AM CET January 23, 2024 9:31:43 AM CET Success
Mounting disk group DATA January 23, 2024 9:31:43 AM CET January 23, 2024 9:31:44 AM CET Success
Mounting disk group RECO January 23, 2024 9:31:53 AM CET January 23, 2024 9:32:01 AM CET Success
Setting ACL for disk groups January 23, 2024 9:32:08 AM CET January 23, 2024 9:32:11 AM CET Success
Register Scan and Vips to Public Network January 23, 2024 9:32:11 AM CET January 23, 2024 9:32:13 AM CET Success
Adding Volume RDOLTTST to Clusterware January 23, 2024 9:32:29 AM CET January 23, 2024 9:32:33 AM CET Success
Adding Volume DATWS12C to Clusterware January 23, 2024 9:32:33 AM CET January 23, 2024 9:32:36 AM CET Success
Adding Volume ACFSCLONE to Clusterware January 23, 2024 9:32:36 AM CET January 23, 2024 9:32:40 AM CET Success
Adding Volume DATLTTST to Clusterware January 23, 2024 9:32:40 AM CET January 23, 2024 9:32:43 AM CET Success
Adding Volume RECO to Clusterware January 23, 2024 9:32:43 AM CET January 23, 2024 9:32:46 AM CET Success
Adding Volume ORAHOME_SH to Clusterware January 23, 2024 9:32:46 AM CET January 23, 2024 9:32:49 AM CET Success
Adding Volume VMSTORE to Clusterware January 23, 2024 9:32:49 AM CET January 23, 2024 9:32:52 AM CET Success
Adding Volume RDOTST23 to Clusterware January 23, 2024 9:32:52 AM CET January 23, 2024 9:32:55 AM CET Success
Adding Volume COMMONSTORE to Clusterware January 23, 2024 9:32:55 AM CET January 23, 2024 9:32:58 AM CET Success
Adding Volume SACD403318 to Clusterware January 23, 2024 9:32:58 AM CET January 23, 2024 9:33:01 AM CET Success
Adding Volume RDOWS12C to Clusterware January 23, 2024 9:33:01 AM CET January 23, 2024 9:33:04 AM CET Success
Adding Volume ODABASE_N0 to Clusterware January 23, 2024 9:33:04 AM CET January 23, 2024 9:33:07 AM CET Success
Adding Volume DATTST23 to Clusterware January 23, 2024 9:33:07 AM CET January 23, 2024 9:33:10 AM CET Success
Adding Volume S2B0A08932 to Clusterware January 23, 2024 9:33:10 AM CET January 23, 2024 9:33:13 AM CET Success
Adding Volume DELPHIXBS to Clusterware January 23, 2024 9:33:13 AM CET January 23, 2024 9:33:16 AM CET Success
Enabling Volume(s) January 23, 2024 9:33:16 AM CET January 23, 2024 9:36:15 AM CET Success
Discover ACFS clones config January 23, 2024 9:39:02 AM CET January 23, 2024 9:39:12 AM CET Success
Configure export clones resource January 23, 2024 9:39:11 AM CET January 23, 2024 9:39:11 AM CET Success
Discover DbHomes ACFS config January 23, 2024 9:39:12 AM CET January 23, 2024 9:39:15 AM CET Success
Discover OraHomeStorage volumes January 23, 2024 9:39:12 AM CET January 23, 2024 9:39:15 AM CET Success
Provisioning service creation January 23, 2024 9:39:17 AM CET January 23, 2024 9:39:17 AM CET Success
Persist new agent state entry January 23, 2024 9:39:17 AM CET January 23, 2024 9:39:17 AM CET Success
Persist new agent state entry January 23, 2024 9:39:17 AM CET January 23, 2024 9:39:17 AM CET Success
Restart DCS Agent January 23, 2024 9:39:17 AM CET January 23, 2024 9:39:19 AM CET SuccessNow my system is restored and a 19.21 GI stack should be running.
Let’s check the ACFS configuration for binaries:
odacli list-dbhome-storages
=============================================================================================================
ID Node Description Disk Group Volume Size(GB) Status
-------------------------------------- ---- -------------------- ---------- ----------- ---------- ----------
d49200b7-5c0a-422b-bab8-2d4392c1fe83 0 ORACLE_HOME DATA orahome_sh 100 CONFIGURED
651512c5-c7b6-4844-883f-a4a31786eba0 0 ORACLE_BASE DATA odabase_n0 10 CONFIGURED
=============================================================================================================
As all DB homes are on ACFS now, no need to manually register them as you would do when using DPR on an old ODA.
Restore the database configuration
The restore-node also has a -d for databases, let’s do it:
odacli restore-node -d
odacli describe-job -i "56e62047-6c95-41e8-a0c1-a52b0f3191af"
Job details
----------------------------------------------------------------
ID: 56e62047-6c95-41e8-a0c1-a52b0f3191af
Description: Restore node service - DB
Status: Success
Created: January 23, 2024 11:47:12 AM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------
Setting up SSH equivalence for 'oracle' January 23, 2024 11:47:15 AM CET January 23, 2024 11:47:17 AM CET Success
Restore BM CPU Pools January 23, 2024 11:47:17 AM CET January 23, 2024 11:47:17 AM CET Success
Register DB home: OraDB19000_home2 January 23, 2024 11:47:17 AM CET January 23, 2024 11:47:20 AM CET Success
Register DB home: OraDB19000_home6 January 23, 2024 11:47:20 AM CET January 23, 2024 11:47:23 AM CET Success
Register DB home: OraDB19000_home4 January 23, 2024 11:47:23 AM CET January 23, 2024 11:47:26 AM CET Success
Register DB home: OraDB12201_home1 January 23, 2024 11:47:27 AM CET January 23, 2024 11:47:50 AM CET Success
Register DB home: OraDB19000_home3 January 23, 2024 11:47:50 AM CET January 23, 2024 11:47:53 AM CET Success
Persist database storage locations January 23, 2024 11:47:53 AM CET January 23, 2024 11:47:53 AM CET Success
Save metadata for DBITST January 23, 2024 11:47:53 AM CET January 23, 2024 11:47:53 AM CET Success
Save metadata for WS12C_SITE1 January 23, 2024 11:47:53 AM CET January 23, 2024 11:47:53 AM CET Success
Save metadata for bsn01 January 23, 2024 11:47:53 AM CET January 23, 2024 11:47:53 AM CET Success
Save metadata for DBWEI January 23, 2024 11:47:53 AM CET January 23, 2024 11:47:53 AM CET Success
Save metadata for MYBRCDB January 23, 2024 11:47:53 AM CET January 23, 2024 11:47:53 AM CET Success
Save metadata for TST23 January 23, 2024 11:47:53 AM CET January 23, 2024 11:47:53 AM CET Success
Save metadata for LTTST January 23, 2024 11:47:53 AM CET January 23, 2024 11:47:53 AM CET Success
Persist database storages January 23, 2024 11:47:53 AM CET January 23, 2024 11:47:53 AM CET Success
Save metadata for DBITST January 23, 2024 11:47:53 AM CET January 23, 2024 11:47:53 AM CET Success
Save metadata for WS12C_SITE1 January 23, 2024 11:47:53 AM CET January 23, 2024 11:47:53 AM CET Success
Save metadata for bsn01 January 23, 2024 11:47:53 AM CET January 23, 2024 11:47:53 AM CET Success
Save metadata for DBWEI January 23, 2024 11:47:53 AM CET January 23, 2024 11:47:53 AM CET Success
Save metadata for MYBRCDB January 23, 2024 11:47:53 AM CET January 23, 2024 11:47:53 AM CET Success
Save metadata for TST23 January 23, 2024 11:47:53 AM CET January 23, 2024 11:47:53 AM CET Success
Save metadata for LTTST January 23, 2024 11:47:53 AM CET January 23, 2024 11:47:53 AM CET Success
Restore database: DBITST January 23, 2024 11:47:53 AM CET January 23, 2024 11:49:14 AM CET Success
+-- Adding database to GI January 23, 2024 11:47:53 AM CET January 23, 2024 11:47:55 AM CET Success
+-- Adding database instance(s) to GI January 23, 2024 11:47:55 AM CET January 23, 2024 11:47:55 AM CET Success
+-- Modifying SPFILE for database January 23, 2024 11:47:55 AM CET January 23, 2024 11:48:31 AM CET Success
+-- Restore password file for database January 23, 2024 11:48:31 AM CET January 23, 2024 11:48:31 AM CET Skipped
+-- Start instance(s) for database January 23, 2024 11:48:31 AM CET January 23, 2024 11:48:52 AM CET Success
+-- Persist metadata for database January 23, 2024 11:48:52 AM CET January 23, 2024 11:48:52 AM CET Success
+-- Create adrci directory January 23, 2024 11:48:52 AM CET January 23, 2024 11:48:52 AM CET Success
+-- Run SqlPatch January 23, 2024 11:48:52 AM CET January 23, 2024 11:49:14 AM CET Success
Restore database: WS12C January 23, 2024 11:49:14 AM CET January 23, 2024 11:50:14 AM CET Success
+-- Adding database to GI January 23, 2024 11:49:14 AM CET January 23, 2024 11:49:16 AM CET Success
+-- Adding database instance(s) to GI January 23, 2024 11:49:16 AM CET January 23, 2024 11:49:16 AM CET Success
+-- Modifying SPFILE for database January 23, 2024 11:49:16 AM CET January 23, 2024 11:49:51 AM CET Success
+-- Restore password file for database January 23, 2024 11:49:51 AM CET January 23, 2024 11:49:51 AM CET Skipped
+-- Start instance(s) for database January 23, 2024 11:49:51 AM CET January 23, 2024 11:50:06 AM CET Success
+-- Persist metadata for database January 23, 2024 11:50:06 AM CET January 23, 2024 11:50:06 AM CET Success
+-- Create adrci directory January 23, 2024 11:50:06 AM CET January 23, 2024 11:50:06 AM CET Success
+-- Run SqlPatch January 23, 2024 11:50:06 AM CET January 23, 2024 11:50:14 AM CET Success
Restore database: bsn01 January 23, 2024 11:50:14 AM CET January 23, 2024 11:51:31 AM CET Success
+-- Adding database to GI January 23, 2024 11:50:14 AM CET January 23, 2024 11:50:16 AM CET Success
+-- Adding database instance(s) to GI January 23, 2024 11:50:16 AM CET January 23, 2024 11:50:16 AM CET Success
+-- Modifying SPFILE for database January 23, 2024 11:50:16 AM CET January 23, 2024 11:50:48 AM CET Success
+-- Restore password file for database January 23, 2024 11:50:48 AM CET January 23, 2024 11:50:48 AM CET Skipped
+-- Start instance(s) for database January 23, 2024 11:50:48 AM CET January 23, 2024 11:51:09 AM CET Success
+-- Persist metadata for database January 23, 2024 11:51:09 AM CET January 23, 2024 11:51:09 AM CET Success
+-- Create adrci directory January 23, 2024 11:51:09 AM CET January 23, 2024 11:51:09 AM CET Success
+-- Run SqlPatch January 23, 2024 11:51:09 AM CET January 23, 2024 11:51:31 AM CET Success
Restore database: DBWEI January 23, 2024 11:51:32 AM CET January 23, 2024 11:52:55 AM CET Success
+-- Adding database to GI January 23, 2024 11:51:32 AM CET January 23, 2024 11:51:33 AM CET Success
+-- Adding database instance(s) to GI January 23, 2024 11:51:33 AM CET January 23, 2024 11:51:33 AM CET Success
+-- Modifying SPFILE for database January 23, 2024 11:51:33 AM CET January 23, 2024 11:52:11 AM CET Success
+-- Restore password file for database January 23, 2024 11:52:11 AM CET January 23, 2024 11:52:11 AM CET Skipped
+-- Start instance(s) for database January 23, 2024 11:52:11 AM CET January 23, 2024 11:52:32 AM CET Success
+-- Persist metadata for database January 23, 2024 11:52:32 AM CET January 23, 2024 11:52:32 AM CET Success
+-- Create adrci directory January 23, 2024 11:52:32 AM CET January 23, 2024 11:52:32 AM CET Success
+-- Run SqlPatch January 23, 2024 11:52:32 AM CET January 23, 2024 11:52:55 AM CET Success
Restore database: MYBRCDB January 23, 2024 11:52:55 AM CET January 23, 2024 11:54:02 AM CET Success
+-- Adding database to GI January 23, 2024 11:52:55 AM CET January 23, 2024 11:52:57 AM CET Success
+-- Adding database instance(s) to GI January 23, 2024 11:52:57 AM CET January 23, 2024 11:52:57 AM CET Success
+-- Modifying SPFILE for database January 23, 2024 11:52:57 AM CET January 23, 2024 11:53:34 AM CET Success
+-- Restore password file for database January 23, 2024 11:53:34 AM CET January 23, 2024 11:53:34 AM CET Skipped
+-- Start instance(s) for database January 23, 2024 11:53:34 AM CET January 23, 2024 11:53:54 AM CET Success
+-- Persist metadata for database January 23, 2024 11:53:54 AM CET January 23, 2024 11:53:54 AM CET Success
+-- Create adrci directory January 23, 2024 11:53:54 AM CET January 23, 2024 11:53:54 AM CET Success
+-- Run SqlPatch January 23, 2024 11:53:54 AM CET January 23, 2024 11:54:02 AM CET Success
Restore database: LTTST January 23, 2024 11:54:02 AM CET January 23, 2024 11:54:53 AM CET Success
+-- Adding database to GI January 23, 2024 11:54:03 AM CET January 23, 2024 11:54:04 AM CET Success
+-- Adding database instance(s) to GI January 23, 2024 11:54:04 AM CET January 23, 2024 11:54:05 AM CET Success
+-- Modifying SPFILE for database January 23, 2024 11:54:05 AM CET January 23, 2024 11:54:30 AM CET Success
+-- Restore password file for database January 23, 2024 11:54:30 AM CET January 23, 2024 11:54:30 AM CET Skipped
+-- Start instance(s) for database January 23, 2024 11:54:30 AM CET January 23, 2024 11:54:45 AM CET Success
+-- Persist metadata for database January 23, 2024 11:54:45 AM CET January 23, 2024 11:54:45 AM CET Success
+-- Create adrci directory January 23, 2024 11:54:45 AM CET January 23, 2024 11:54:45 AM CET Success
+-- Run SqlPatch January 23, 2024 11:54:45 AM CET January 23, 2024 11:54:53 AM CET Success
Restore Object Stores January 23, 2024 11:54:53 AM CET January 23, 2024 11:54:53 AM CET Success
Remount NFS backups January 23, 2024 11:54:53 AM CET January 23, 2024 11:54:53 AM CET Success
Restore BackupConfigs January 23, 2024 11:54:53 AM CET January 23, 2024 11:54:53 AM CET Success
Reattach backupconfigs to DBs January 23, 2024 11:54:53 AM CET January 23, 2024 11:54:53 AM CET Success
Restore backup reports January 23, 2024 11:54:53 AM CET January 23, 2024 11:54:53 AM CET Success
odacli list-databases
ID DB Name DB Type DB Version CDB Class Edition Shape Storage Status DB Home ID
---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
7390837e-3d0f-4457-9e18-8a06fc5be359 DBITST SI 19.20.0.0.230718 true OLTP SE odb1 ASM CONFIGURED f8998897-34f2-435c-9898-710eb960e361
3d689dde-b7d7-4d75-b04e-d1ed77fbebfd WS12C SI 19.20.0.0.230718 false OLTP EE odb1 ACFS CONFIGURED 80aaed91-d1ff-42c5-be16-590120c7a7f2
3d187e6a-0381-454b-8034-7545821106ff bsn01 SI 19.19.0.0.230418 true OLTP SE odb2 ASM CONFIGURED d8aa26c6-be94-4a79-a653-48095961d9a2
6b2d8aaa-825f-40c0-96c4-0bbc81d84e28 DBWEI SI 19.20.0.0.230718 true OLTP EE odb1 ASM CONFIGURED 80aaed91-d1ff-42c5-be16-590120c7a7f2
95426fd1-5434-4c99-9704-194735d84b33 MYBRCDB SI 19.20.0.0.230718 false OLTP EE odb1 ASM CONFIGURED 80aaed91-d1ff-42c5-be16-590120c7a7f2
f836307f-5dae-4a0b-bb86-0752e1b1948c LTTST SI 19.20.0.0.230718 false OLTP SE odb1 ACFS CONFIGURED f8998897-34f2-435c-9898-710eb960e361Databases are now restored and available.
Note that it only works for bare metal databases. For now, there is no VMs and no DB Systems:
odacli list-vms
No data found for VM
odacli list-dbsystems
No data found for DB SystemRestore the VMs and DB Systems
There are some specifics commands for VMs and DB Systems restore:
odacli restore-node -kvm
odacli describe-job -i "6704768e-d931-4e87-931d-37562bd61f04"
Job details
----------------------------------------------------------------
ID: 6704768e-d931-4e87-931d-37562bd61f04
Description: Restore node service - KVM
Status: Success
Created: January 23, 2024 11:57:08 AM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------
Validate backup files January 23, 2024 11:57:08 AM CET January 23, 2024 11:57:08 AM CET Success
Read backup metadata January 23, 2024 11:57:08 AM CET January 23, 2024 11:57:08 AM CET Success
Check existing resources January 23, 2024 11:57:08 AM CET January 23, 2024 11:57:08 AM CET Success
Create ACFS mount point January 23, 2024 11:57:08 AM CET January 23, 2024 11:57:08 AM CET Success
Register ACFS resources January 23, 2024 11:57:08 AM CET January 23, 2024 11:57:10 AM CET Success
Restore VM Storages metadata January 23, 2024 11:57:10 AM CET January 23, 2024 11:57:10 AM CET Success
Restore VDisks metadata January 23, 2024 11:57:10 AM CET January 23, 2024 11:57:10 AM CET Success
Restore CPU Pools January 23, 2024 11:57:10 AM CET January 23, 2024 11:57:10 AM CET Success
Restore VNetworks January 23, 2024 11:57:10 AM CET January 23, 2024 11:57:16 AM CET Success
Patch VM's domain config files January 23, 2024 11:57:16 AM CET January 23, 2024 11:57:24 AM CET Success
Restore VMs January 23, 2024 11:57:24 AM CET January 23, 2024 11:57:26 AM CET Success
Restore VMs metadata January 23, 2024 11:57:26 AM CET January 23, 2024 11:57:26 AM CET Success
Start VMs January 23, 2024 11:57:26 AM CET January 23, 2024 11:57:30 AM CET Success
odacli restore-node -dbs
odacli describe-job -i "b720c6dc-7f46-4373-9359-40f8bafe777f"
Job details
----------------------------------------------------------------
ID: b720c6dc-7f46-4373-9359-40f8bafe777f
Description: Restore node service - DBSYSTEM
Status: Success
Created: January 23, 2024 11:58:30 AM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------
Validate DB System json files January 23, 2024 11:58:31 AM CET January 23, 2024 11:58:31 AM CET Success
Process DB Systems for restore operation January 23, 2024 11:58:31 AM CET January 23, 2024 11:58:31 AM CET Success
Create DB System ACFS mount points January 23, 2024 11:58:31 AM CET January 23, 2024 11:58:31 AM CET Success
Patch libvirt xml for DB Systems January 23, 2024 11:58:31 AM CET January 23, 2024 11:58:35 AM CET Success
Restore DB System Networks January 23, 2024 11:58:35 AM CET January 23, 2024 11:58:35 AM CET Success
Add DB Systems to Clusterware January 23, 2024 11:58:35 AM CET January 23, 2024 11:58:36 AM CET Success
Validate start dependencies January 23, 2024 11:58:36 AM CET January 23, 2024 11:58:36 AM CET Success
Start DB Systems January 23, 2024 11:58:36 AM CET January 23, 2024 11:58:42 AM CET Success
Wait for DB Systems bootstrap January 23, 2024 11:58:42 AM CET January 23, 2024 11:59:51 AM CET Success
Export clones repository for DB January 23, 2024 11:59:51 AM CET January 23, 2024 11:59:51 AM CET Success
Systems post restore
Export ASM client cluster config on BM January 23, 2024 11:59:51 AM CET January 23, 2024 11:59:56 AM CET Success
Import ASM client cluster config to January 23, 2024 11:59:56 AM CET January 23, 2024 12:01:46 PM CET Success
OLR (within DB Systems)
Import ASM client cluster config to January 23, 2024 12:01:46 PM CET January 23, 2024 12:04:36 PM CET Success
OCR (within DB Systems)
Update DB System status to after January 23, 2024 12:04:36 PM CET January 23, 2024 12:04:37 PM CET Success
restore operation
odacli list-vms
Name VM Storage Current State Target State Created Updated
-------------------- -------------------- --------------- --------------- ------------------------ ------------------------
ubuntu02 VMstore ONLINE ONLINE 2023-12-07 15:06:57 CET 2024-01-23 11:57:26 CET
VMMAW01 VMstore ONLINE ONLINE 2023-12-07 14:51:32 CET 2024-01-23 11:57:26 CET
delphix VMstore ONLINE ONLINE 2024-01-03 15:49:57 CET 2024-01-23 11:57:26 CET
ubuntu01 VMstore ONLINE ONLINE 2023-12-07 15:05:54 CET 2024-01-23 11:57:26 CET
odacli list-dbsystems
Name Shape Cores Memory GI version DB version Status Created Updated
-------------------- ---------- ----- ---------- ------------------ ------------------ --------------- ------------------------ ------------------------
testdbs02 odb2 2 16.00 GB 19.20.0.0.230718 N/A CONFIGURED 2024-01-04 13:47:20 CET 2024-01-23 12:04:36 CET
testdbs01 odb2 2 16.00 GB 19.20.0.0.230718 19.20.0.0.230718 CONFIGURED 2024-01-04 12:07:05 CET 2024-01-23 12:04:37 CETMy VMs are back to life. Fine!
Patching the DB homes
Once your ODA is fully restored on top of the new OS version, next step is to update the DB homes/databases. Nothing new here compared to the previous versions. You need to download, unzip and register the new DB clone first:
unzip p30403662_1921000_Linux-x86-64.zip
odacli update-repository -f /opt/dbi/odacli-dcs-19.21.0.0.0-231211-DB-19.21.0.0.zip
sleep 30; odacli describe-job -i "169a08ae-897a-49e4-b9f9-3db22c9d8940"
Job details
----------------------------------------------------------------
ID: 169a08ae-897a-49e4-b9f9-3db22c9d8940
Description: Repository Update
Status: Success
Created: January 24, 2024 9:30:01 AM CET
Message: /opt/dbi/odacli-dcs-19.21.0.0.0-231211-DB-19.21.0.0.zip
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------
Unzip bundle January 24, 2024 9:30:01 AM CET January 24, 2024 9:30:39 AM CET SuccessYou will need to choose between patching at the DB home level or database level:
odacli list-dbhomes
ID Name DB Version DB Edition Home Location Status
---------------------------------------- -------------------- ------------------------------ ---------- ------------------------------------------------------- ----------
80aaed91-d1ff-42c5-be16-590120c7a7f2 OraDB19000_home4 19.20.0.0.230718 EE /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4 CONFIGURED
a684a7cc-1ebd-425c-90df-31fad25c3295 OraDB12201_home1 12.2.0.1.220118 SE /u01/app/odaorahome/oracle/product/12.2.0.1/dbhome_1 CONFIGURED
aa3cab94-8a6d-4230-9aa9-8c517ce24836 OraDB19000_home6 19.20.0.0.230718 EE /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_6 CONFIGURED
d8aa26c6-be94-4a79-a653-48095961d9a2 OraDB19000_home2 19.19.0.0.230418 SE /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2 CONFIGURED
f8998897-34f2-435c-9898-710eb960e361 OraDB19000_home3 19.20.0.0.230718 SE /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_3 CONFIGURED
odacli list-databases
ID DB Name DB Type DB Version CDB Class Edition Shape Storage Status DB Home ID
---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
7390837e-3d0f-4457-9e18-8a06fc5be359 DBITST SI 19.20.0.0.230718 true OLTP SE odb1 ASM CONFIGURED f8998897-34f2-435c-9898-710eb960e361
3d689dde-b7d7-4d75-b04e-d1ed77fbebfd WS12C SI 19.20.0.0.230718 false OLTP EE odb1 ACFS CONFIGURED 80aaed91-d1ff-42c5-be16-590120c7a7f2
3d187e6a-0381-454b-8034-7545821106ff bsn01 SI 19.19.0.0.230418 true OLTP SE odb2 ASM CONFIGURED d8aa26c6-be94-4a79-a653-48095961d9a2
6b2d8aaa-825f-40c0-96c4-0bbc81d84e28 DBWEI SI 19.20.0.0.230718 true OLTP EE odb1 ASM CONFIGURED 80aaed91-d1ff-42c5-be16-590120c7a7f2
95426fd1-5434-4c99-9704-194735d84b33 MYBRCDB SI 19.20.0.0.230718 false OLTP EE odb1 ASM CONFIGURED 80aaed91-d1ff-42c5-be16-590120c7a7f2
f836307f-5dae-4a0b-bb86-0752e1b1948c LTTST SI 19.20.0.0.230718 false OLTP SE odb1 ACFS CONFIGURED f8998897-34f2-435c-9898-710eb960e361I usually work at the DB home level. Let’s patch the first DB home here:
odacli create-prepatchreport -d -i f8998897-34f2-435c-9898-710eb960e361 -v 19.21.0.0.0
sleep 600; odacli describe-prepatchreport -i 74b70078-c1cd-41b0-8980-f61b40b75504
Patch pre-check report
------------------------------------------------------------------------
Job ID: 74b70078-c1cd-41b0-8980-f61b40b75504
Description: Patch pre-checks for [DB, ORACHKDB]: DbHome is OraDB19000_home3
Status: FAILED
Created: January 24, 2024 9:31:54 AM CET
Result: One or more pre-checks failed for [ORACHK]
Node Name
---------------
dbi-oda1
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__DB__
Validate DB Home ID Success Validated DB Home ID:
f8998897-34f2-435c-9898-710eb960e361
Validate patching tag Success Validated patching tag: 19.21.0.0.0.
Is system provisioned Success Verified system is provisioned
Validate minimum agent version Success Validated minimum agent version
Is GI upgraded Success Validated GI is upgraded
Validate available space for Success Validated free space required under
db /u01
Validate dbHomesOnACFS Success User has configured disk group for
configured Database homes on ACFS
Validate Oracle base Success Successfully validated Oracle Base
Is DB clone available Success Successfully validated clone file
exists
Evaluate DBHome patching with Success Successfully validated updating
RHP dbhome with RHP. and local patching
is possible
Validate command execution Success Validated command execution
__ORACHK__
Running orachk Failed ORAchk validation failed: .
Validate command execution Success Validated command execution
Verify the Fast Recovery Area Failed AHF-2929: FRA space management
(FRA) has reclaimable space problem file types are present
without an RMAN backup completion
within the last 7 days
Verify the Fast Recovery Area Failed AHF-2929: FRA space management
(FRA) has reclaimable space problem file types are present
without an RMAN backup completion
within the last 7 daysSometimes ORACHK will show failurea for different things, but your database may accept some different settings or status ORACHK doesn’t like. So, at some point, you may need to force the DB home update with -f.
odacli update-dbhome -i f8998897-34f2-435c-9898-710eb960e361 -v 19.21.0.0.0
DCS-10227:cannot proceed; prepatchreport has errors for version: 19.21.0.0.0, components: orachk, on node: dbi-oda1
odacli update-dbhome -i f8998897-34f2-435c-9898-710eb960e361 -v 19.21.0.0.0 -f
sleep 600; odacli describe-job -i "34e41b4e-b845-4f44-b73b-429fec2d84f5"
odacli describe-job -i "34e41b4e-b845-4f44-b73b-429fec2d84f5"
Job details
----------------------------------------------------------------
ID: 34e41b4e-b845-4f44-b73b-429fec2d84f5
Description: DB Home Patching: Home ID is f8998897-34f2-435c-9898-710eb960e361
Status: Success
Created: January 24, 2024 10:05:55 AM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------
Creating wallet for DB Client January 24, 2024 10:06:48 AM CET January 24, 2024 10:06:48 AM CET Success
Patch databases by RHP - [DBITST] January 24, 2024 10:06:48 AM CET January 24, 2024 10:12:10 AM CET Success
Updating database metadata January 24, 2024 10:12:10 AM CET January 24, 2024 10:12:10 AM CET Success
Set log_archive_dest for Database January 24, 2024 10:12:10 AM CET January 24, 2024 10:12:13 AM CET Success
Set target_pdbs parameter January 24, 2024 10:12:13 AM CET January 24, 2024 10:13:12 AM CET Success
Patch databases by RHP - [LTTST] January 24, 2024 10:13:12 AM CET January 24, 2024 10:17:09 AM CET Success
Updating database metadata January 24, 2024 10:17:09 AM CET January 24, 2024 10:17:09 AM CET Success
Set log_archive_dest for Database January 24, 2024 10:17:09 AM CET January 24, 2024 10:17:11 AM CET Success
Update System version January 24, 2024 10:17:11 AM CET January 24, 2024 10:17:11 AM CET Success
Generating and saving BOM January 24, 2024 10:17:11 AM CET January 24, 2024 10:18:52 AM CET Success
TDE parameter update January 24, 2024 10:20:07 AM CET January 24, 2024 10:20:07 AM CET Success
Let’s check if a new DB home has been provisioned and if my 2 databases are using this new home:
odacli list-dbhomes
ID Name DB Version DB Edition Home Location Status
---------------------------------------- -------------------- ------------------------------ ---------- ------------------------------------------------------- ----------
80aaed91-d1ff-42c5-be16-590120c7a7f2 OraDB19000_home4 19.20.0.0.230718 EE /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4 CONFIGURED
a684a7cc-1ebd-425c-90df-31fad25c3295 OraDB12201_home1 12.2.0.1.220118 SE /u01/app/odaorahome/oracle/product/12.2.0.1/dbhome_1 CONFIGURED
aa3cab94-8a6d-4230-9aa9-8c517ce24836 OraDB19000_home6 19.20.0.0.230718 EE /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_6 CONFIGURED
d8aa26c6-be94-4a79-a653-48095961d9a2 OraDB19000_home2 19.19.0.0.230418 SE /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2 CONFIGURED
f8998897-34f2-435c-9898-710eb960e361 OraDB19000_home3 19.20.0.0.230718 SE /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_3 CONFIGURED
a248cb7f-8bb7-459d-95e3-09081a71991d OraDB19000_home7 19.21.0.0.231017 SE /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_7 CONFIGURED
odacli list-databases | grep 19.21
7390837e-3d0f-4457-9e18-8a06fc5be359 DBITST SI 19.21.0.0.231017 true OLTP SE odb1 ASM CONFIGURED a248cb7f-8bb7-459d-95e3-09081a71991d
f836307f-5dae-4a0b-bb86-0752e1b1948c LTTST SI 19.21.0.0.231017 false OLTP SE odb1 ACFS CONFIGURED a248cb7f-8bb7-459d-95e3-09081a71991dEverything is fine!
Don’t forget that the previous DB home will not be deleted unless you do it manually, maybe later.
Cleanse the old patches
There is no need to cleanse the old patches as the system was reimaged.
Post-patching tasks
You will need to put back your specific configuration as your ODA was reimaged. Depending on your configuration, it may be few simple tasks or a more complex setup.
The tasks could be:
- add your additional tools if needed
- create additional users and groups
- put back folder and files
- put back your profile scripts for grid and oracle users
- configure monitoring again…
Patching DB Systems
If you use DB Systems on your ODA, meaning that some of your databases are running in dedicated VMs, you will need to apply the patch inside each DB System. It will use the same “reimaging” principle: you won’t need an ISO but the system disk of your VMs will be replaced by the brand new VM image running on Linux 8.8. Your system settings will be lost, for example your additional users, packages, folders. Make sure to backup everything before patching each DB System.
First, download, unzip and register the new 19.21 DB System template on your ODA:
unzip p32451228_1921000_Linux-x86-64.zip
odacli update-repository -f /opt/dbi/odacli-dcs-19.21.0.0.0-ODAVM.zip
odacli describe-job -i "011509ff-6022-4420-8e10-706b3cf804d1"
Job details
----------------------------------------------------------------
ID: 011509ff-6022-4420-8e10-706b3cf804d1
Description: Repository Update
Status: Success
Created: January 23, 2024 6:03:57 PM CET
Message: /opt/dbi/odacli-dcs-19.21.0.0.0-ODAVM.zip
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------
Unzip bundle January 23, 2024 6:03:57 PM CET January 23, 2024 6:06:09 PM CET SuccessList the DB Systems running on your appliance:
odacli list-dbsystems
Name Shape Cores Memory GI version DB version Status Created Updated
-------------------- ---------- ----- ---------- ------------------ ------------------ --------------- ------------------------ ------------------------
testdbs02 odb2 2 16.00 GB 19.20.0.0.230718 N/A CONFIGURED 2024-01-04 13:47:20 CET 2024-01-23 12:04:36 CET
testdbs01 odb2 2 16.00 GB 19.20.0.0.230718 19.20.0.0.230718 CONFIGURED 2024-01-04 12:07:05 CET 2024-01-23 12:04:37 CET
I will patch testdbs01 in this example. You don’t need to connect to the DB Systems: operations are done on the ODA itself. There is a prepatch report to create before appyling the patch:
odacli create-preupgradereport -dbs testdbs01
odacli describe-preupgradereport -i 33fc4317-272c-4cd9-9a73-f7204253cb06
Upgrade pre-check report
------------------------------------------------------------------------
Job ID: 33fc4317-272c-4cd9-9a73-f7204253cb06
Description: Run pre-upgrade checks for DB System: testdbs01
Status: SUCCESS
Created: January 23, 2024 6:06:27 PM CET
Result: All pre-checks succeeded
Node Name
---------------
dbi-oda1
Check Status Message Action
------------------------------ -------- -------------------------------------- --------------------------------------
__DBSYSTEM__
Validate DB System State Success DB System 'testdbs01' is in None
'CONFIGURED' state
Verify existence of DBVM image Success DB System image version '19.21.0.0.0' None
is present in repository
Verify existence of Database Success Database clone version None
clone '19.20.0.0.230718' is present in
repository
Verify existence of GI clone Success GI clone version '19.21.0.0.231017' None
is present in repository
Node Name
---------------
testdbs01
Check Status Message Action
------------------------------ -------- -------------------------------------- --------------------------------------
__OS__
Check Required OS files Success All the required files are present None
Check Additional OS RPMs Success No RPMs outside of base ISO were None
found on the system
__SYS__
Validate Hardware Type Success Current hardware is supported None
Validate System Version Success System version 19.20.0.0.0 is None
supported
Verify System Timezone Success Succesfully verified the time zone None
file
Verify Grid User Success Grid user is verified None
Verify Grid Version Success Oracle Grid Infrastructure is running None
on the '19.20.0.0.230718' version on
all nodes
Check Audit Files Alert Audit files found under These files will be lost after
/u01/app/oracle/audit reimage. Backup the audit files to a
location outside the ODA system
Verify number of Databases Success Only one database is active None
Verify number of Database Success Only one database home is configured None
Homes
__DB__
Validate Database Status Success Database 'DBTST1' is running and is None
in 'CONFIGURED' state
Validate Database Version Success Version '19.20.0.0.230718' for None
database 'DBTST1' is supported
Validate Database Datapatch Success Database 'DBTST1' is completely None
Application Status applied with datapatch
Validate TDE wallet presence Success Database 'DBTST1' is not TDE enabled. None
Skipping TDE wallet presence check.
Validate Database Home Success Database home location check passed None
location for database DBTST1
__CERTIFICATES__
Check using custom Success Using Default key pair None
certificates
Check the agent of the DB Success All the agents of the DB systems are None
System accessible accessible
Obviously, audit files will be lost as they reside on the OS filesystems. On my DB System, I also have additional system users, folders and scripts. They will be lost without any warning.
As the prepatch is OK, let’s patch this DB System:
odacli upgrade-dbsystem -n testdbs01
Enter password for system testdbs01:
Retype password for system testdbs01:
sleep 1800 ; odacli describe-job -i 7abe8461-2247-4243-ab9c-9278ffb45594
Job details
----------------------------------------------------------------
ID: 7abe8461-2247-4243-ab9c-9278ffb45594
Description: DB System testdbs01 upgrade
Status: Success
Created: January 23, 2024 6:18:58 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------
Run DB System upgrade prechecks January 23, 2024 6:18:58 PM CET January 23, 2024 6:19:33 PM CET Success
Save provisioning payload January 23, 2024 6:19:33 PM CET January 23, 2024 6:19:33 PM CET Success
Set DB System as detaching January 23, 2024 6:19:33 PM CET January 23, 2024 6:19:33 PM CET Success
Detach node - DPR January 23, 2024 6:19:33 PM CET January 23, 2024 6:23:15 PM CET Success
Copy server archive file from DB System January 23, 2024 6:23:15 PM CET January 23, 2024 6:23:16 PM CET Success
Set DB System as recreating January 23, 2024 6:23:16 PM CET January 23, 2024 6:23:16 PM CET Success
Remove DB System from Clusterware January 23, 2024 6:23:16 PM CET January 23, 2024 6:23:22 PM CET Success
Delete ASM client cluster config January 23, 2024 6:23:22 PM CET January 23, 2024 6:23:28 PM CET Success
Deprovision DB System VM(s) January 23, 2024 6:23:28 PM CET January 23, 2024 6:23:29 PM CET Success
Delete DB System ACFS filesystem January 23, 2024 6:23:29 PM CET January 23, 2024 6:23:32 PM CET Success
Delete DB System ACFS mount point January 23, 2024 6:23:32 PM CET January 23, 2024 6:23:32 PM CET Success
Delete DB System ASM volume January 23, 2024 6:23:32 PM CET January 23, 2024 6:23:37 PM CET Success
Delete imported certificates January 23, 2024 6:23:37 PM CET January 23, 2024 6:23:38 PM CET Success
Delete DB System metadata January 23, 2024 6:23:38 PM CET January 23, 2024 6:23:38 PM CET Success
Load provisioning payload January 23, 2024 6:23:38 PM CET January 23, 2024 6:23:38 PM CET Success
Validate DB System prerequisites January 23, 2024 6:23:38 PM CET January 23, 2024 6:23:42 PM CET Success
Create DB System metadata January 23, 2024 6:23:42 PM CET January 23, 2024 6:23:44 PM CET Success
Create DB System ASM volume January 23, 2024 6:23:44 PM CET January 23, 2024 6:23:52 PM CET Success
Create DB System ACFS mount point January 23, 2024 6:23:52 PM CET January 23, 2024 6:23:52 PM CET Success
Create DB System ACFS filesystem January 23, 2024 6:23:52 PM CET January 23, 2024 6:24:00 PM CET Success
Create DB System VM ACFS snapshots January 23, 2024 6:24:00 PM CET January 23, 2024 6:24:36 PM CET Success
Create temporary SSH key pair January 23, 2024 6:24:36 PM CET January 23, 2024 6:24:37 PM CET Success
Create DB System cloud-init config January 23, 2024 6:24:37 PM CET January 23, 2024 6:24:38 PM CET Success
Provision DB System VM(s) January 23, 2024 6:24:38 PM CET January 23, 2024 6:24:42 PM CET Success
Attach disks to DB System January 23, 2024 6:24:42 PM CET January 23, 2024 6:24:42 PM CET Success
Add DB System to Clusterware January 23, 2024 6:24:42 PM CET January 23, 2024 6:24:43 PM CET Success
Start DB System January 23, 2024 6:24:43 PM CET January 23, 2024 6:24:47 PM CET Success
Wait DB System VM first boot January 23, 2024 6:24:47 PM CET January 23, 2024 6:25:48 PM CET Success
Setup Mutual TLS (mTLS) January 23, 2024 6:25:48 PM CET January 23, 2024 6:26:07 PM CET Success
Export clones repository January 23, 2024 6:26:07 PM CET January 23, 2024 6:26:07 PM CET Success
Setup ASM client cluster config January 23, 2024 6:26:07 PM CET January 23, 2024 6:26:09 PM CET Success
Copy ASM client cluster config January 23, 2024 6:26:09 PM CET January 23, 2024 6:26:10 PM CET Success
Install DB System January 23, 2024 6:26:10 PM CET January 23, 2024 6:43:04 PM CET Success
Copy server archive file to DB System January 23, 2024 6:43:04 PM CET January 23, 2024 6:43:05 PM CET Success
Unpack server archive zip file January 23, 2024 6:43:05 PM CET January 23, 2024 6:43:11 PM CET Success
Cleanup temporary SSH key pair January 23, 2024 6:43:11 PM CET January 23, 2024 6:43:11 PM CET Success
Set DB System as reconfiguring January 23, 2024 6:43:11 PM CET January 23, 2024 6:43:11 PM CET Success
Change Database file ownership January 23, 2024 6:43:11 PM CET January 23, 2024 6:43:16 PM CET Success
Restore node - DPR January 23, 2024 6:43:16 PM CET January 23, 2024 6:48:21 PM CET Success
Set upgraded DB System as configured January 23, 2024 6:48:21 PM CET January 23, 2024 6:48:21 PM CET SuccessIt took about 30 minutes to patch one DB System.
Let’s check OS version and database version on it:
ssh [email protected]
cat /etc/oracle-release
Oracle Linux Server release 8.8
odacli list-databases
ID DB Name DB Type DB Version CDB Class Edition Shape Storage Status DB Home ID
---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
1d42028d-3f5f-4f0f-af9f-966e8a627363 DBTST1 SI 19.20.0.0.230718 false OLTP EE odb2 ASM CONFIGURED e21938ca-474a-4892-a806-26261e34ff36
exit
My OS is Linux 8.8, and my database is still using 19.20, as expected. It’s now possible to update the database to 19.21 like you did for bare metal databases, if you need to.
Conclusion
This patch release is the first one using Data Preserving Reprovisioning for X7/X8/X9. It means that you’ll get a cleaner system after applying this patch, but it also means that there are more risks during patching as you will replace the system with a fresh Linux setup. At some point, the restore-node may hang, and there is no possibility to revert back to the old system. Make sure to protect your databases with Data Guard or Dbvisit Standby.
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/JDU_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/ALN_web-min-scaled.jpg)
Ata Ur Rehman
29.03.2024/opt/oracle/dcs/bin/odacli create-prepatchreport -v 19.22.0.0.0 -os
DCS-10290:Operation: Create Prepatchreport not allowed: operating system upgrade is required
Please reply me.
Ata