Jérôme Dubar, auteur/autrice sur dbi Blog

Reduce downtime when refreshing your non-production databases using Multitenant

Jérôme Dubar — Thu, 21 May 2026 09:55:25 +0000

Introduction

You probably refresh your non-production Oracle databases with production data from time to time or on a regular basis. Without Multitenant, the most common procedure to do this refresh is a DUPLICATE FROM BACKUP with RMAN. The drawback is the unavailability of the database being refreshed during the DUPLICATE. You first need to remove the old version of the database, then start the DUPLICATE and wait until it’s finished. If you have Enterprise Edition and enough CPU, you can lower the time needed for the refresh by allocating a sufficient number of channels. But with a small number of CPU (which is normal for a non-production server), or eventually with Standard Edition (single channel RMAN operations only), a multi-TB database refresh can take several hours to complete. And if it fails for some reasons, you need to retry the refresh, extending even more the downtime.

Multitenant brought new possibilities for refreshing a database, and my favorite one is a CREATE PLUGGABLE DATABASE from a database link (DB link). It’s dead easy compared to a DUPLICATE FROM BACKUP on a non-CDB database. And you can lower the downtime to the very minimum. Here is how I did this for several projects.

How to lower the downtime to the minimum when refreshing a non-production PDB?

You probably know that one of the advantage of a pluggable database is the easiness of changing its name. You just need to stop the PDB, rename it, and restart it. You can then use this technique to refresh a PDB under a temporary name and let the actual PDB available during the refresh. Once the refresh is finished, drop or rename the actual PDB, and rename the newest one to its target name. Even if your refresh takes hours, your downtime is limited to a couple of seconds/minutes.

Step 1: add an additional grant for source PDB’s administrator

The PDB administrator on the source database must have the CREATE PLUGGABLE DATABASE privilege:

ssh oracle@p01-srv-ora
. oraenv <<< P19PMT
sqlplus / as sysdba
Alter session set container=P19_ERP;
grant create pluggable database to SYSERP;
exit

Step 2: add a TNS entry on the target server

The target server must have a TNS entry to the source PDB (production). If your source PDB and its container are protected by a Data Guard configuration, dont’t forget to add both addresses:

ssh root@t01-srv-ora
su – oracle
. oraenv <<< D19PMT
vi $ORACLE_HOME/network/admin/tnsnames.ora
…
P19_ERP =
(DESCRIPTION =
   (LOAD_BALANCE = OFF)
   (FAILOVER = ON)
   (ADDRESS_LIST =
     (ADDRESS = (PROTOCOL = TCP)(HOST = p01-srv-ora)(PORT = 1521))
     (ADDRESS = (PROTOCOL = TCP)(HOST = p02-srv-ora)(PORT = 1521))
   )
   (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = P19_ERP)
   )
)

tnsping P19_ERP
…

Step 3: create a DB link on the target container

A DB link is required on the target container:

ssh root@t01-srv-ora
su – oracle
. oraenv <<< D19PMT
sqlplus / as sysdba
CREATE DATABASE LINK P19_ERP CONNECT TO SYSERP IDENTIFIED BY "*************"  USING 'P19_ERP';

select count(*) from dual@P19_ERP;
  COUNT(*)
----------
         1
exit

Step 4: prepare the scripts for the refresh

Basically, refresh will have 5 main tasks:

create a new PDB with a temporary name _NEW on the target container from the source PDB
start the new PDB for its correct registration in the container
run an optional script for modifying production data (masking, disabling tasks, …)
stop and rename the current PDB to _OLD, then start it again
stop and rename the new PDB to its target name and start it again

Task 2 is needed because you cannot rename a PDB immediately after creation. You first need to open it, then close it for being able to change its name.

Let’s create 2 scripts on the target server, one shell script and one SQL script:

vi /home/oracle/scripts/refresh_D19_ERP.sh
#!/bin/bash
export ORACLE_SID=D19PMT
export REFRESH_LOG=/home/oracle/scripts/log/refresh_D19_ERP_`date +%d_%m_%Y-%H_%M_%S`.log
export ORACLE_HOME=`cat /etc/oratab | grep $ORACLE_SID | awk -F ':' '{print $2;}'`
date >> $REFRESH_LOG
$ORACLE_HOME/bin/sqlplus / as sysdba @/home/oracle/scripts/refresh_D19_ERP.sql >> $REFRESH_LOG
date >> $REFRESH_LOG
exit 0

vi /home/oracle/scripts/refresh_D19_ERP.sql
set timing on
show pdbs
alter pluggable database D19_ERP_OLD close immediate;
Drop pluggable database D19_ERP_OLD including datafiles;
show pdbs
create pluggable database D19_ERP_NEW from P19_ERP@P19_ERP ;
show pdbs
alter pluggable database D19_ERP_NEW open;
show pdbs
alter session set container=D19_ERP_NEW;
@/home/oracle/scripts/post_refresh_D19_ERP.sql 
alter session set container=CDB$ROOT;
alter pluggable database D19_ERP close immediate;
alter pluggable database D19_ERP rename global_name to D19_ERP_OLD;
alter pluggable database D19_ERP_OLD open;
show pdbs
alter pluggable database D19_ERP_NEW close immediate;
alter pluggable database D19_ERP_NEW rename global_name to D19_ERP;
Alter pluggable database D19_ERP open;
Alter pluggable database D19_ERP save state;
show pdbs
exit

It does the job, although these are very basic scripts: further controls could be added to trap errors, manage services, and so on.

Step 5: schedule the refresh

Scheduling can be done through the crontab, for example every evening at 11.30PM:

crontab -l | grep D19_ERP | grep refresh
30 23 * * * sh /home/oracle/scripts/refresh_D19_ERP.sh

Conclusion

This is definitely a smart solution as soon as you have enough space on disk to have 2 copies of the PDB. It’s quite reliable and ticks all the boxes where I deployed these scripts.

L’article Reduce downtime when refreshing your non-production databases using Multitenant est apparu en premier sur dbi Blog.

Not enough space for using ODABR? Reduce /u01 and /opt size on your ODA

Jérôme Dubar — Fri, 24 Apr 2026 06:42:11 +0000

Introduction

The patching of an Oracle Database Appliance needs to be secured. This is because it won’t limit to patching the databases: there is plenty of other components embedded with the patch, the goal being to keep everything updated. A possible rollback may be needed.

For sure, there is no possible rollback for firmwares, BIOS and ILOM (it’s unlikely you would need to go back to previous version), but you may need to rollback when it comes to patching the OS and the GI stack. Unlike DB homes, OS and GI stack are located on the ODA local disks, and can be protected during patching with LVM snapshots. This is the purpose of ODABR: making LVM snapshots of /, /u01 and /opt before applying the patch. If something goes wrong, you can revert to the previous stable state.

ODABR will need enough space on local disks for creating the snapshots, and sometimes available free space is not enough. How to deal with that?

When taking the snapshots?

Before using ODABR, I would recommend doing a cleanup of the following filesystems: /, /u01 and /opt. You will need 20+% of free space in these filesystems, otherwise the patch prechecks won’t give you the green light. ODABR is based on Copy-On-Write technology, meaning that old versions of the changed blocks will be kept for going back in time if needed. Don’t unzip the patch files after taking the snapshots if the patch files reside on the local disks! Once you’re ready to patch, the minute before registering the patch and applying the DCS components update, you can use ODABR to take the snapshots.

When releasing the snapshots?

Once you successfully applied the system patch and GI patch, and once you verified that everything runs fine with the new GI binaries, you can safely use ODABR to delete the snapshots. If you need to apply multiple patches, make new snapshots before each jump, you will never need to revert to the oldest version once you are in a stable intermediate version.

Are the ODABR snapshots useful for DB homes?

Absolutely not. DB homes are located under /u01/app/odaorahome, a dedicated ACFS filesystem on the DATA diskgroup. You don’t need snapshot features on this very filesystem because patching a DB home will create a new home and move the database into this new DB home, the old one being kept as long as you want to. You can delete older DB homes months after patching was done. I usually delete old ones as part of the patching job once everything is fine, but I always backup the old DB home with a tar czf in case of. If you need to revert a database to an old DB home, you can do it manually and use a RMAN backup to restore the database, or eventually rollback the datapatch if needed. This is rarely used.

Location of the snapshots

ODABR relies on LVM snapshots, not something really specific to ODA. Snapshots are stored in the remaining space of the Volume Group. Most ODAs were sold with 2x 500GB local disks (SSDs) protected with a software RAID and allocated to a single Volume Group VolGroupSys. When doing the ODA setup, only a part of the storage is allocated to the system volumes as Logical Volumes (LogVolSwap, LogVolOpt, LogVolU01, LogVolRoot), meaning that a comfortable amount of space is available for ODABR:

pvs
  PV           VG          Fmt  Attr PSize   PFree
  /dev/md126p3 VolGroupSys lvm2 a--  446.09g 285.09g

/opt/odabr/odabr backup -snap
...
SUCCESS: 2026-04-22 17:10:33: LVM snapshots backup done successfully

pvs
  PV           VG          Fmt  Attr PSize   PFree
  /dev/md126p3 VolGroupSys lvm2 a--  446.09g 150.09g

ODABR additionally backups ASM metadata, because if you revert to a previous set of LVM snapshots with an older GI release, your ASM metadata would also need to be in the corresponding version.

Default snapshot sizes and real needs

Default snapshots size is the size of the Logical Volumes, here is an example for a 30GB /, a 55GB /opt and a 50GB /u01:

/opt/odabr/odabr infosnap

│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
 odabr - ODA node Backup Restore - Version: 2.0.2-06
 Copyright 2013, 2025, Oracle and/or its affiliates.
 --------------------------------------------------------
 RACPack, Cloud Innovation and Solution Engineering Team
│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│

LVM snap name         status                COW Size              Data%
-------------         ----------            ----------            ------
root_snap             active                30.00 GiB             0.01%
opt_snap              active                55.00 GiB             0.01%
u01_snap              active                50.00 GiB             0.01%

This is quite confortable, there won’t be 100% blocks changed during the patching, for sure. If you don’t have these 135GB of free space, you can specify lower values for the snapshots, but make sure to limit modifications on the files residing on these 3 Logical Volumes during the patch:

/opt/odabr/odabr backup -snap -osize 18 -rsize 10 -usize 25

These settings were enough for my latest patches.

The ODA X9-2 series exception

As far as I remember, ODAs always had a confortable system disk sizes. 500GB is large enough for OS, odacli and GI stack plus a couple of files of your own. Since version 19.10, DB homes and diagnostic destination are now located within ACFS volumes, meaning that it doesn’t take a single MB from the local disks:

df -h | grep -e Filesystem -e u01
Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/VolGroupSys-LogVolU01    49G   27G   20G  58% /u01
/dev/asm/odabase_n0-225              30G  2.5G   28G   9% /u01/app/odaorabase0
/dev/asm/orahome_sh-225              80G   44G   37G  54% /u01/app/odaorahome

As 19.10 was already available when X9-2 was released, Oracle decided to decrease the system disks from 500GB to 250GB, meaning that you didn’t have the margin you had with the previous generations. The problem was solved with X10 series and onwards, system disks are now back to 500GB as they were on X8-2 series.

What I mean is that you’ll be most probably concerned about filesystem contention for ODABR if you have X9-2 series.

How can you deal with insufficient disk space when using ODABR?

Imagine you have this kind of configuration:

pvs
  PV           VG          Fmt  Attr PSize   PFree
  /dev/md126p3 VolGroupSys lvm2 a--  222.56g 2.56g

On this ODA, /opt and /u01 have been extended for some reasons, and there is almost no space left on the Volume Group.

If you try to use ODABR to take snapshots, you will get this error:

Available LVM size X is less than required snapshot size Y

2.56GB is way too small for using ODABR: you will need to reduce the size of /u01, /opt or both. First, a cleanup is needed to free up some space. Once done, you’ll be able to reduce the size of /u01 and/or /opt. But these filesystems will need to be unmounted. And it cannot be done without stopping/starting some processes. Actually, pretty much all the processes running on your ODA.

Stopping the processes and reducing the filesystems

Let’s reduce the /u01 and /opt Logical Volumes’ size in this example:

su - root

df -h /u01 /opt

Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/VolGroupSys-LogVolU01    61G   23G   36G  39% /u01
/dev/mapper/VolGroupSys-LogVolOpt    70G   48G   22G  68% /opt

export ORACLE_HOME=/u01/app/19.30.0.0/grid
$ORACLE_HOME/bin/crsctl stop crs

lvreduce -L 50G /dev/VolGroupSys/LogVolU01 -r
Do you want to unmount "/u01" ? [Y|n] y
umount: /u01: target is busy.
fsadm: Cannot proceed with mounted filesystem "/u01".
  /usr/sbin/fsadm failed: 1
  Filesystem resize failed.

lsof /u01/
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
java    23727 root   26r   REG  252,3 16303574 1208968 /u01/app/19.30.0.0/grid/jlib/srvm.jar

kill -9 23727 ; umount /u01
umount: /u01: target is busy.

kill -9 23727 ; umount /u01
-bash: kill: (23727) - No such process

lvreduce -L 50G /dev/VolGroupSys/LogVolU01 -r
fsck from util-linux 2.32.1
/dev/mapper/VolGroupSys-LogVolU01: 76718/4063232 files (7.3% non-contiguous), 6220430/16252928 blocks
resize2fs 1.46.2 (28-Feb-2021)
Resizing the filesystem on /dev/mapper/VolGroupSys-LogVolU01 to 13107200 (4k) blocks.
The filesystem on /dev/mapper/VolGroupSys-LogVolU01 is now 13107200 (4k) blocks long.

  Size of logical volume VolGroupSys/LogVolU01 changed from 62.00 GiB (1984 extents) to 50.00 GiB (1600 extents).
  Logical volume VolGroupSys/LogVolU01 successfully resized.


systemctl stop initdcsagent
systemctl stop initdcscontroller
systemctl stop initdcsadmin
systemctl stop oda-mysql
systemctl stop oracle-ODA_DCS-ODA_DCS0
systemctl stop oracle-tfa
systemctl stop oracle-ohasd


lsof /opt
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
bash       6697 root  cwd    DIR  252,2     4096 1703937 /opt/dbi
OSWatcher  7799 grid  cwd    DIR  252,2     4096 2235031 /opt/oracle/dcs/oracle.ahf/data/repository/suptools/dbioda01/oswbb/grid/oswbb
OSWatcher  7799 grid    1w   REG  252,2     4703 2228894 /opt/oracle/dcs/oracle.ahf/data/repository/suptools/dbioda01/oswbb/grid/run_1773762195.log (deleted)
OSWatcher  7799 grid    2w   REG  252,2     4703 2228894 /opt/oracle/dcs/oracle.ahf/data/repository/suptools/dbioda01/oswbb/grid/run_1773762195.log (deleted)
OSWatcher  7799 grid  255r   REG  252,2    65286 2235047 /opt/oracle/dcs/oracle.ahf/data/repository/suptools/dbioda01/oswbb/grid/oswbb/OSWatcher.sh
OSWatcher  9903 grid  cwd    DIR  252,2     4096 2235031 /opt/oracle/dcs/oracle.ahf/data/repository/suptools/dbioda01/oswbb/grid/oswbb
OSWatcher  9903 grid    1w   REG  252,2     4703 2228894 /opt/oracle/dcs/oracle.ahf/data/repository/suptools/dbioda01/oswbb/grid/run_1773762195.log (deleted)
OSWatcher  9903 grid    2w   REG  252,2     4703 2228894 /opt/oracle/dcs/oracle.ahf/data/repository/suptools/dbioda01/oswbb/grid/run_1773762195.log (deleted)
OSWatcher  9903 grid  255r   REG  252,2     8035 2235046 /opt/oracle/dcs/oracle.ahf/data/repository/suptools/dbioda01/oswbb/grid/oswbb/OSWatcherFM.sh
sleep     56757 grid  cwd    DIR  252,2     4096 2235031 /opt/oracle/dcs/oracle.ahf/data/repository/suptools/dbioda01/oswbb/grid/oswbb
sleep     56757 grid    1w   REG  252,2     4703 2228894 /opt/oracle/dcs/oracle.ahf/data/repository/suptools/dbioda01/oswbb/grid/run_1773762195.log (deleted)
sleep     56757 grid    2w   REG  252,2     4703 2228894 /opt/oracle/dcs/oracle.ahf/data/repository/suptools/dbioda01/oswbb/grid/run_1773762195.log (deleted)
sleep     56835 grid  cwd    DIR  252,2     4096 2235031 /opt/oracle/dcs/oracle.ahf/data/repository/suptools/dbioda01/oswbb/grid/oswbb
sleep     56835 grid    1w   REG  252,2     4703 2228894 /opt/oracle/dcs/oracle.ahf/data/repository/suptools/dbioda01/oswbb/grid/run_1773762195.log (deleted)
sleep     56835 grid    2w   REG  252,2     4703 2228894 /opt/oracle/dcs/oracle.ahf/data/repository/suptools/dbioda01/oswbb/grid/run_1773762195.log (deleted)

kill -9 7799 9903


lvreduce -L 55G /dev/VolGroupSys/LogVolOpt -r
Do you want to unmount "/opt" ? [Y|n] y
fsck from util-linux 2.32.1
/dev/mapper/VolGroupSys-LogVolOpt: Inode 655911 extent tree (at level 1) could be narrower.  IGNORED.
/dev/mapper/VolGroupSys-LogVolOpt: 57911/4587520 files (2.4% non-contiguous), 12785629/18350080 blocks
resize2fs 1.46.2 (28-Feb-2021)
Resizing the filesystem on /dev/mapper/VolGroupSys-LogVolOpt to 14417920 (4k) blocks.
The filesystem on /dev/mapper/VolGroupSys-LogVolOpt is now 14417920 (4k) blocks long.

  Size of logical volume VolGroupSys/LogVolOpt changed from 70.00 GiB (2240 extents) to 55.00 GiB (1760 extents).
  Logical volume VolGroupSys/LogVolOpt successfully resized.

As I don’t want to manually restart the various processes, and as I usually do a sanity reboot before patching, let’s reboot the server. Everything will then be properly started after.

reboot
...

df -h /u01 /opt
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/VolGroupSys-LogVolU01   49G   23G   25G  49% /u01
/dev/mapper/VolGroupSys-LogVolOpt   54G   48G  4.0G  93% /opt

Further recommendations

You don’t really need a lot of space on the system disks. Put your patch files and other specific files on a NFS volume shared across your ODAs, and keep the system disks to their default’s size to keep a confortable margin for ODABR, especially on ODA X9-2 series. Eventually, allow a few more GB to /u01 and/or /opt but keep at least 70GB free for ODABR snapshots. Anticipate the patching and test ODABR several days before, it does not cost anything, and snapshots can be removed immediately without any downtime. Worst case is having to patch now and discovering that there is no space left on disks for using ODABR.

Not being able to secure your patch with ODABR is the second NOGO for patching, the first one being a faulty component reported by odaadmcli show server.

Conclusion

If you don’t have enough free space for ODABR snapshots, postpone the patching. ODABR is a mandatory safety net. Reducing /u01 and /opt filesystems’s size is possible. It will require stopping everything, but as you will patch your ODA, you’ve already allocated a sufficient downtime.

L’article Not enough space for using ODABR? Reduce /u01 and /opt size on your ODA est apparu en premier sur dbi Blog.

How to patch your ODA to 19.30?

Jérôme Dubar — Fri, 06 Mar 2026 15:10:54 +0000

Introduction

Patch 19.30 is now available for Oracle Database Appliance series. Let’s find out what’s new and how to apply this patch.

What’s new?

The real new feature is the possibility to rollback a server component patch done by an odacli update-servercomponents. Until now, in the rare cases you would need to do a rollback, you could only rely on ODABR. And guess what? This new rollback feature makes use of ODABR. So keep doing manual ODABR snapshots prior attempting to patch your ODA.

Regarding the odacli update-dcscomponents, it now has a dedicated job engine you can query with a new command odacli describe-admin-job to see the progress. This is useful because this job now lasts longer. It’s always good to know the involved steps and their status of such a process.

This version is 19.30, meaning that bare metal GI stack is still using 19c binaries. 23/26ai databases are still limited to DB Systems, meaning that bare metal databases are limited to 19c.

As you can guess, this patch is mainly a bundle patch for security and bug fixes. A bigger update is expected later this year.

Which ODA is compatible with this 19.30 release?

The latest ODAs X11-HA, X11-L and X11-S are supported, as well as X10, X9-2 and X8-2 series. X7-2 series and older ones are not supported anymore. If you own one from these older generations, you will not be able to patch it anymore. If you’re using X8-2 ODAs, available from late 2019 to mid 2022, the last patch is planned for August 2027.

I still recommend keeping your ODA 7 years, not less, not more. This blog post is still relevant today: https://www.dbi-services.com/blog/why-you-should-consider-keeping-your-oda-more-than-5-years /.

Is this patch a cumulative one?

The rule is now well established: you can apply a patch on top of the four previous ones. 19.30 can then be applied on top of 19.29, 19.28, 19.27 and 19.26. Patching once a year will prevent having to apply 2 or more patches, meaning a longer downtime.

What’s new since 19.29 is the additional monthly system patch. This is a special patch for system only, for those who cannot wait for the global patch to be released. First one was 19.29.0.1.0. 19.30 can be applied on top of it.

In my lab at dbi services, I will use an ODA X8-2M running 19.29 with one DB home, one database and one DB System. The DB System is already running a 26ai database. The given patching procedure should be the same if you come from 19.26 or later.

Is there also a patch for my databases?

Only databases version 19c are supported on bare metal. Patch for 19c is obviously 19.30. For a 26ai database running inside a DB System, you will patch from 23.26.0.0 to 23.26.1.0. This is how the new version numbering works now.

Download the patch and clone files

These files are mandatory:

38776074 => the patch itself
30403673 => the GI clone needed for deploying newer 19c GI version
30403662 => the DB clone for deploying newer version of 19c

These files are optional:

30403643 => ISO file for reimaging, not needed for patching
36524660 => System image for deploying a new 26ai DB System
36524627 => the GI clone needed for deploying/patching to newer 26ai GI version
36524642 => the DB clone for deploying/patching to newer 26ai version
32451228 => The newer system image for 19c DB Systems
38776071 => The patch for 26ai DB Systems

Be sure to choose the very latest 19.30 when downloading some files, patch number is the same for older versions of GI clones, DB clones and ISO files.

Prepare the patching

Before starting, please check these prerequisites:

filesystems /, /opt, /u01 and /root have at least 20% of available free space
unzip the downloaded patch files
additional manually installed rpms must be removed
revert profile scripts to default’s one (for grid and oracle users)
make sure you’ve planned a sufficient downtime (4+ hours depending on the number of databases and DB Systems)
do a sanity reboot before patching to kill zombie processes
use ODABR to make snapshots of the important filesystems prior patching: this tool is now included in the software distribution

df -h / /u01 /opt/
Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/VolGroupSys-LogVolRoot   30G   13G   16G  46% /
/dev/mapper/VolGroupSys-LogVolU01    59G   23G   34G  40% /u01
/dev/mapper/VolGroupSys-LogVolOpt    69G   34G   33G  52% /opt

cd /opt/dbi
for a in `ls p*1930*.zip` ; do unzip -o $a ; rm -f $a ; done

reboot
...

pvs
  PV           VG          Fmt  Attr PSize   PFree
  /dev/md126p3 VolGroupSys lvm2 a--  446.09g 260.09g

/opt/odabr/odabr backup --snap
INFO: 2026-02-26 11:29:54: Please check the logfile '/opt/odabr/out/log/odabr_24079.log' for more details


│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
 odabr - ODA node Backup Restore - Version: 2.0.2-05
 Copyright 2013, 2025, Oracle and/or its affiliates.
 --------------------------------------------------------
 RACPack, Cloud Innovation and Solution Engineering Team
│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│

INFO: 2026-02-26 11:29:54: Checking superuser
INFO: 2026-02-26 11:29:54: Checking Bare Metal

INFO: 2026-02-26 11:29:54: Removing existing LVM snapshots
WARNING: 2026-02-26 11:29:54: LVM snapshot for 'opt' does not exist
WARNING: 2026-02-26 11:29:54: LVM snapshot for 'u01' does not exist
WARNING: 2026-02-26 11:29:54: LVM snapshot for 'root' does not exist

INFO: 2026-02-26 11:29:54: Checking current OS version...
INFO: 2026-02-26 11:29:54: Checking LVM restore backgroud process
INFO: 2026-02-26 11:29:54: Checking LVM size

INFO: 2026-02-26 11:29:54: Boot device backup
INFO: 2026-02-26 11:29:54: Getting EFI device
INFO: 2026-02-26 11:29:54: ...step1 - unmounting EFI
INFO: 2026-02-26 11:29:54: ...step2 - making efi device backup
SUCCESS: 2026-02-26 11:29:57: ...EFI device backup saved as '/opt/odabr/out/hbi/efi.img'
INFO: 2026-02-26 11:29:57: ...step3 - checking EFI device backup
INFO: 2026-02-26 11:29:57: Getting boot device
INFO: 2026-02-26 11:29:57: ...step1 - making boot device backup using tar
SUCCESS: 2026-02-26 11:30:03: ...boot content saved as '/opt/odabr/out/hbi/boot.tar.gz'
INFO: 2026-02-26 11:30:03: ...step2 - unmounting boot
INFO: 2026-02-26 11:30:04: ...step3 - making boot device backup using dd
SUCCESS: 2026-02-26 11:30:09: ...boot device backup saved as '/opt/odabr/out/hbi/boot.img'
INFO: 2026-02-26 11:30:09: ...step4 - mounting boot
INFO: 2026-02-26 11:30:09: ...step5 - mounting EFI
INFO: 2026-02-26 11:30:10: ...step6 - checking boot device backup

INFO: 2026-02-26 11:30:10: Making OCR physical backup
INFO: 2026-02-26 11:30:11: ...ocr backup saved as '/opt/odabr/out/hbi/ocrbackup_24079.bck'
SUCCESS: 2026-02-26 11:30:11: OCR physical backup created successfully
INFO: 2026-02-26 11:30:11: OCR export backup
INFO: 2026-02-26 11:30:12: ...ocr export saved as '/opt/odabr/out/hbi/ocrexport_24079.bck'
SUCCESS: 2026-02-26 11:30:12: OCR export backup created successfully
INFO: 2026-02-26 11:30:12: Saving clusterware patch level as '/opt/odabr/out/hbi/clusterware_patch_level.info'
SUCCESS: 2026-02-26 11:30:12: Clusterware patch level saved successfully

INFO: 2026-02-26 11:30:12: Making LVM snapshot backup
SUCCESS: 2026-02-26 11:30:13: ...snapshot backup for 'opt' created successfully
SUCCESS: 2026-02-26 11:30:14: ...snapshot backup for 'u01' created successfully
SUCCESS: 2026-02-26 11:30:14: ...snapshot backup for 'root' created successfully
SUCCESS: 2026-02-26 11:30:14: LVM snapshots backup done successfully

Version precheck

Start to check the current version of the various components:

odacli describe-component
System Version
--------------
19.29.0.0.0

System Node Name
----------------
dbioda01

Local System Version
--------------------
19.29.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                      19.29.0.0.0          up-to-date

GI                                       19.29.0.0.251021     up-to-date

DB {
     OraDB19000_home9                    19.29.0.0.251021     up-to-date
     [CPROD19]
}

DCSCONTROLLER                            19.29.0.0.0          up-to-date

DCSCLI                                   19.29.0.0.0          up-to-date

DCSAGENT                                 19.29.0.0.0          up-to-date

DCSADMIN                                 19.29.0.0.0          up-to-date

OS                                       8.10                 up-to-date

ILOM                                     5.1.5.22.r165351     up-to-date

BIOS                                     52160100             up-to-date

LOCAL CONTROLLER FIRMWARE {
     [c4]                                8000D9AB             up-to-date
}

SHARED CONTROLLER FIRMWARE {
     [c0, c1]                            VDV1RL06             up-to-date
}

LOCAL DISK FIRMWARE {
     [c2d0, c2d1]                        XC311132             up-to-date
}

HMP                                      2.4.10.1.600         up-to-date

List the DB homes, databases, DB Systems and VMs:

odacli list-dbhomes
ID                                       Name                 DB Version           DB Edition Home Location                                            Status
---------------------------------------- -------------------- -------------------- ---------- -------------------------------------------------------- ----------
57c0dd7f-dcf4-4a38-9e79-4bf8c78e81bb     OraDB19000_home9     19.29.0.0.251021     EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_9     CONFIGURED


odacli list-databases
ID                                       DB Name    DB Type  DB Version           CDB     Class    Edition  Shape    Storage  Status       DB Home ID
---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
976a80f2-4653-469f-8cd4-ddc1a21aff51     CPROD19    SI       19.29.0.0.251021     true    OLTP     EE       odb8     ASM      CONFIGURED   57c0dd7f-dcf4-4a38-9e79-4bf8c78e81bb


odacli list-dbsystems
Name                  Shape       GI version          DB info                         Status                  Created                   Updated
--------------------  ----------  ------------------  ------------------------------  ----------------------  ------------------------  ------------------------
dbs-04-tst            dbs2        23.26.0.0.0         23.26(CONFIGURED=1)             CONFIGURED              2026-01-12 10:14:46 CET   2026-01-12 10:45:45 CET

odacli list-vms
No data found for resource VM.

Update the DCS components

Updating DCS components is the first step, after registering the patch file:

odacli update-repository -f /opt/dbi/oda-sm-19.30.0.0.0-260210-server.zip
sleep 30 ; odacli describe-job -i e3ba068f-01db-45c3-949d-b79f43c8d6b7
Job details
----------------------------------------------------------------
                     ID:  e3ba068f-01db-45c3-949d-b79f43c8d6b7
            Description:  Repository Update
                 Status:  Success
                Created:  February 26, 2026 11:30:48 CET
                Message:  /opt/dbi/oda-sm-19.30.0.0.0-260210-server.zip

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Unzip bundle                             February 26, 2026 11:30:49 CET           February 26, 2026 11:31:10 CET           Success


odacli describe-component
System Version
--------------
19.29.0.0.0

System Node Name
----------------
dbioda01

Local System Version
--------------------
19.29.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                      19.29.0.0.0          19.30.0.0.0

GI                                       19.29.0.0.251021     19.30.0.0.260120

DB {
     OraDB19000_home9                    19.29.0.0.251021     19.30.0.0.260120
     [CPROD19]
}

DCSCONTROLLER                            19.29.0.0.0          19.30.0.0.0

DCSCLI                                   19.29.0.0.0          19.30.0.0.0

DCSAGENT                                 19.29.0.0.0          19.30.0.0.0

DCSADMIN                                 19.29.0.0.0          19.30.0.0.0

OS                                       8.10                 up-to-date

ILOM                                     5.1.5.22.r165351     5.1.5.29.r167438

BIOS                                     52160100             52170100

LOCAL CONTROLLER FIRMWARE {
     [c4]                                8000D9AB             up-to-date
}

SHARED CONTROLLER FIRMWARE {
     [c0, c1]                            VDV1RL06             up-to-date
}

LOCAL DISK FIRMWARE {
     [c2d0, c2d1]                        XC311132             up-to-date
}

HMP                                      2.4.10.1.600         up-to-date

Let’s update the DCS components to 19.30:

odacli update-dcsadmin -v 19.30.0.0.0

sleep 90 ; odacli describe-job -i "c3c278ad-89e6-4c7d-b9fd-27833a187e43"
Job details
----------------------------------------------------------------
                     ID:  c3c278ad-89e6-4c7d-b9fd-27833a187e43
            Description:  DcsAdmin patching to 19.30.0.0.0
                 Status:  Success
                Created:  February 26, 2026 11:32:11 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Dcs-admin upgrade                        February 26, 2026 11:32:11 CET           February 26, 2026 11:32:21 CET           Success
Ping DCS Admin                           February 26, 2026 11:32:21 CET           February 26, 2026 11:33:29 CET           Success


sleep 30 ;  odacli update-dcscomponents -v 19.30.0.0.0
sleep 300 ; odacli describe-admin-job -i 2aeda3f3-df4d-4f7c-a0ce-b57eeab0448b
Job details
----------------------------------------------------------------
                     ID:  2aeda3f3-df4d-4f7c-a0ce-b57eeab0448b
            Description:  Update-dcscomponents to 19.30.0.0.0
                 Status:  Success
                Created:  February 26, 2026 11:34:51 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Pre-checks for update DCS components     February 26, 2026 11:34:57 CET           February 26, 2026 11:35:05 CET           Success
Update DCS components                    February 26, 2026 11:35:05 CET           February 26, 2026 11:35:05 CET           Success
Stop DCS-Agent                           February 26, 2026 11:35:05 CET           February 26, 2026 11:35:05 CET           Success
Update MySql                             February 26, 2026 11:35:05 CET           February 26, 2026 11:35:42 CET           Success
Apply metadata schema changes            February 26, 2026 11:35:42 CET           February 26, 2026 11:35:42 CET           Success
Modify MySQL Metadata                    February 26, 2026 11:35:42 CET           February 26, 2026 11:35:43 CET           Success
Update DCS-Agent                         February 26, 2026 11:35:43 CET           February 26, 2026 11:35:57 CET           Success
Update DCS-Cli                           February 26, 2026 11:35:57 CET           February 26, 2026 11:35:59 CET           Success
Update DCS-Controller                    February 26, 2026 11:35:59 CET           February 26, 2026 11:36:22 CET           Success
Update AHF RPM                           February 26, 2026 11:36:22 CET           February 26, 2026 11:38:41 CET           Success
Reset Keystore password                  February 26, 2026 11:38:41 CET           February 26, 2026 11:39:02 CET           Success
Update HAMI                              February 26, 2026 11:39:02 CET           February 26, 2026 11:39:54 CET           Success
Remove old library files                 February 26, 2026 11:39:54 CET           February 26, 2026 11:39:54 CET           Success
Post DCS update actions                  February 26, 2026 11:39:54 CET           February 26, 2026 11:39:54 CET           Success

System patching

Let’s do the prepatching of the system:

odacli create-prepatchreport -sc -v 19.30.0.0.0

sleep 180 ; odacli describe-prepatchreport -i c06cd4d0-d30c-4063-a3c1-3b86db6625b0

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  c06cd4d0-d30c-4063-a3c1-3b86db6625b0
            Description:  Patch pre-checks for [OS, ILOM, ORACHKSERVER, SERVER] to 19.30.0.0.0
                 Status:  SUCCESS
                Created:  February 26, 2026 11:40:17 AM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
dbioda01

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is patch location available     Success   Patch location is available.
Verify All OS patches           Success   No dependencies found for RPMs being
                                          removed, updated and installed. Check
                                          /opt/oracle/dcs/log/jobfiles/
                                          dnfdryrunout_2026-02-26_11-40-
                                          34.0688_236.log file for more details
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate all KVM ACFS           Success   All KVM ACFS resources are running
resources are running
Validate DB System VM states    Success   All DB System VMs states are expected
Validate DB System AFD state    Success   All DB Systems are on required
                                          versions
Validate command execution      Success   Validated command execution

OK let’s apply the system patch:

odacli update-servercomponents -v 19.30.0.0.0
...

The server will reboot at the end of the patching (it took 40 minutes on my X8-2M). Let’s then check the job:

odacli describe-job -i "8d5b4b63-3a92-46e4-b466-9fd46cdf8b3a"
Job details
----------------------------------------------------------------
                     ID:  8d5b4b63-3a92-46e4-b466-9fd46cdf8b3a
            Description:  Server Patching to 19.30.0.0.0
                 Status:  Success
                Created:  February 26, 2026 11:42:47 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Validating GI user metadata              February 26, 2026 11:42:53 CET           February 26, 2026 11:42:54 CET           Success
Modify BM udev rules                     February 26, 2026 11:42:54 CET           February 26, 2026 11:43:05 CET           Success
Validate ILOM server reachable           February 26, 2026 11:42:54 CET           February 26, 2026 11:42:54 CET           Success
Stop oakd                                February 26, 2026 11:43:05 CET           February 26, 2026 11:43:08 CET           Success
Creating local repository                February 26, 2026 11:43:09 CET           February 26, 2026 11:43:11 CET           Success
OSPatchBaseRepo
Updating versionlock plugin              February 26, 2026 11:43:11 CET           February 26, 2026 11:43:14 CET           Success
Applying OS Patches                      February 26, 2026 11:43:14 CET           February 26, 2026 11:50:26 CET           Success
Applying HMP Patches                     February 26, 2026 11:50:27 CET           February 26, 2026 11:50:30 CET           Success
Creating local repository HMPPatchRepo   February 26, 2026 11:50:27 CET           February 26, 2026 11:50:27 CET           Success
Oda-hw-mgmt upgrade                      February 26, 2026 11:50:31 CET           February 26, 2026 11:51:02 CET           Success
Patch location validation                February 26, 2026 11:50:31 CET           February 26, 2026 11:50:31 CET           Success
Setting SELinux mode                     February 26, 2026 11:50:31 CET           February 26, 2026 11:50:31 CET           Success
Installing SQLcl software                February 26, 2026 11:51:02 CET           February 26, 2026 11:51:06 CET           Success
OSS Patching                             February 26, 2026 11:51:02 CET           February 26, 2026 11:51:02 CET           Success
Applying Firmware local Disk Patches     February 26, 2026 11:51:06 CET           February 26, 2026 11:51:10 CET           Success
Applying Firmware local Controller Patch February 26, 2026 11:51:10 CET           February 26, 2026 11:51:14 CET           Success
Applying Firmware shared Controller      February 26, 2026 11:51:15 CET           February 26, 2026 11:51:19 CET           Success
Patch
Checking Ilom patch Version              February 26, 2026 11:51:19 CET           February 26, 2026 11:51:19 CET           Success
Patch location validation                February 26, 2026 11:51:19 CET           February 26, 2026 11:51:19 CET           Success
Disabling IPMI v2                        February 26, 2026 11:51:20 CET           February 26, 2026 11:51:21 CET           Success
Save password in Wallet                  February 26, 2026 11:51:20 CET           February 26, 2026 11:51:20 CET           Success
Apply Ilom patch                         February 26, 2026 11:51:21 CET           February 26, 2026 12:02:07 CET           Success
Copying Flash Bios to Temp location      February 26, 2026 12:02:07 CET           February 26, 2026 12:02:07 CET           Success
Start oakd                               February 26, 2026 12:02:08 CET           February 26, 2026 12:02:24 CET           Success
Add SYSNAME in Env                       February 26, 2026 12:02:25 CET           February 26, 2026 12:02:25 CET           Success
Cleanup JRE Home                         February 26, 2026 12:02:25 CET           February 26, 2026 12:02:25 CET           Success
Starting the clusterware                 February 26, 2026 12:02:25 CET           February 26, 2026 12:03:59 CET           Success
Update lvm.conf file                     February 26, 2026 12:04:00 CET           February 26, 2026 12:04:00 CET           Success
Generating and saving BOM                February 26, 2026 12:04:01 CET           February 26, 2026 12:04:34 CET           Success
Update System full patch version         February 26, 2026 12:04:01 CET           February 26, 2026 12:04:01 CET           Success
Update System rebootless patch version   February 26, 2026 12:04:01 CET           February 26, 2026 12:04:01 CET           Success
PreRebootNode Actions                    February 26, 2026 12:04:34 CET           February 26, 2026 12:07:10 CET           Success
Reboot Node                              February 26, 2026 12:07:10 CET           February 26, 2026 12:18:17 CET           Success

GI patching

Let’s register the patch file, and do the precheck for GI:

odacli update-repository -f /opt/dbi/odacli-dcs-19.30.0.0.0-260210-GI-19.30.0.0.zip
sleep 70 ; odacli describe-job -i "5699a201-8f50-499c-98a8-18b2e79ca356"

Job details
----------------------------------------------------------------
                     ID:  5699a201-8f50-499c-98a8-18b2e79ca356
            Description:  Repository Update
                 Status:  Success
                Created:  February 26, 2026 12:32:04 CET
                Message:  /opt/dbi/odacli-dcs-19.30.0.0.0-260210-GI-19.30.0.0.zip

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Unzip bundle                             February 26, 2026 12:32:05 CET           February 26, 2026 12:33:14 CET           Success

odacli create-prepatchreport -gi -v 19.30.0.0.0

sleep 180 ; odacli describe-prepatchreport -i ac157412-cfe2-4b2e-ae04-e7082cd4014f

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  ac157412-cfe2-4b2e-ae04-e7082cd4014f
            Description:  Patch pre-checks for [RHPGI, GI] to 19.30.0.0.0
                 Status:  SUCCESS
                Created:  February 26, 2026 12:34:00 PM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
dbioda01

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__RHPGI__
Validate available space        Success   Validated free space under /u01
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Successfully validated minimum version
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM is online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Validate space for clones       Success   Clones volume is already created
volume
Validate command execution      Success   Validated command execution

Let’s apply the GI update now:

odacli update-gihome -v 19.30.0.0.0
sleep 500 ;  odacli describe-job -i "857bb637-82ec-4de9-a820-5ab9b895e9f8"

Job details
----------------------------------------------------------------
                     ID:  857bb637-82ec-4de9-a820-5ab9b895e9f8
            Description:  Patch GI with RHP to 19.30.0.0.0
                 Status:  Success
                Created:  February 26, 2026 12:38:29 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Patch GI with RHP to 19.30.0.0.0         February 26, 2026 12:38:44 CET           February 26, 2026 12:46:00 CET           Success
Registering image                        February 26, 2026 12:38:45 CET           February 26, 2026 12:38:45 CET           Success
Registering working copy                 February 26, 2026 12:38:45 CET           February 26, 2026 12:38:46 CET           Success
Starting the clusterware                 February 26, 2026 12:38:45 CET           February 26, 2026 12:38:45 CET           Success
Creating GI home directories             February 26, 2026 12:38:46 CET           February 26, 2026 12:38:46 CET           Success
Extract GI clone                         February 26, 2026 12:38:46 CET           February 26, 2026 12:38:46 CET           Success
Provisioning Software Only GI with RHP   February 26, 2026 12:38:46 CET           February 26, 2026 12:38:46 CET           Success
Registering image                        February 26, 2026 12:38:46 CET           February 26, 2026 12:38:46 CET           Success
Patch GI with RHP                        February 26, 2026 12:39:20 CET           February 26, 2026 12:45:25 CET           Success
Set CRS ping target                      February 26, 2026 12:45:25 CET           February 26, 2026 12:45:25 CET           Success
Updating .bashrc                         February 26, 2026 12:45:25 CET           February 26, 2026 12:45:26 CET           Success
Updating GI home metadata                February 26, 2026 12:45:26 CET           February 26, 2026 12:45:26 CET           Success
Updating GI home version                 February 26, 2026 12:45:26 CET           February 26, 2026 12:45:31 CET           Success
Updating All DBHome version              February 26, 2026 12:45:31 CET           February 26, 2026 12:45:36 CET           Success
Starting the clusterware                 February 26, 2026 12:45:56 CET           February 26, 2026 12:45:56 CET           Success
Validate ACFS resources are running      February 26, 2026 12:45:56 CET           February 26, 2026 12:45:57 CET           Success
Validate GI availability                 February 26, 2026 12:45:56 CET           February 26, 2026 12:45:56 CET           Success
Validate DB System VMs states            February 26, 2026 12:45:57 CET           February 26, 2026 12:45:58 CET           Success
Patch CPU Pools distribution             February 26, 2026 12:45:58 CET           February 26, 2026 12:45:58 CET           Success
Patch DB System domain config            February 26, 2026 12:45:58 CET           February 26, 2026 12:45:58 CET           Success
Patch KVM CRS type                       February 26, 2026 12:45:58 CET           February 26, 2026 12:45:58 CET           Success
Patch VM vDisks CRS dependencies         February 26, 2026 12:45:58 CET           February 26, 2026 12:45:58 CET           Success
Save custom VNetworks to storage         February 26, 2026 12:45:58 CET           February 26, 2026 12:45:59 CET           Success
Add network filters to DB Systems        February 26, 2026 12:45:59 CET           February 26, 2026 12:46:00 CET           Success
Create network filters                   February 26, 2026 12:45:59 CET           February 26, 2026 12:45:59 CET           Success
Patch DB Systems custom scale metadata   February 26, 2026 12:46:00 CET           February 26, 2026 12:46:00 CET           Success
Patch DB Systems vDisks CRS dependencies February 26, 2026 12:46:00 CET           February 26, 2026 12:46:00 CET           Success

No reboot is needed for this patch.

Check the versions

odacli describe-component
System Version
--------------
19.30.0.0.0

System Node Name
----------------
dbioda01

Local System Version
--------------------
19.30.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                      19.30.0.0.0          up-to-date

GI                                       19.30.0.0.260120     up-to-date

DB {
     OraDB19000_home9                    19.29.0.0.251021     19.30.0.0.260120
     [CPROD19]
}

DCSCONTROLLER                            19.30.0.0.0          up-to-date

DCSCLI                                   19.30.0.0.0          up-to-date

DCSAGENT                                 19.30.0.0.0          up-to-date

DCSADMIN                                 19.30.0.0.0          up-to-date

OS                                       8.10                 up-to-date

ILOM                                     5.1.5.29.r167438     up-to-date

BIOS                                     52170100             up-to-date

LOCAL CONTROLLER FIRMWARE {
     [c4]                                8000D9AB             up-to-date
}

SHARED CONTROLLER FIRMWARE {
     [c0, c1]                            VDV1RL06             up-to-date
}

LOCAL DISK FIRMWARE {
     [c2d0, c2d1]                        XC311132             up-to-date
}

HMP                                      2.4.10.1.600         up-to-date

Patching the storage

Patching the storage is only needed if describe-component tells you that you’re not up-to-date. On my X8-2M, it wasn’t needed. If your ODA need the storage patch, it’s easy:

odacli update-storage -v 19.30.0.0.0
odacli describe-job -i ...

The server will reboot once done.

Patching the DB homes

It’s now time to patch the DB home and the database on my ODA. Let’s first unzip and register the patch file in the repository:

odacli update-repository -f /opt/dbi/odacli-dcs-19.30.0.0.0-260210-DB-19.30.0.0.zip 
sleep 60; odacli describe-job -i 5690c811-9030-427e-82f1-caeeba236329

Job details
----------------------------------------------------------------
                     ID:  5690c811-9030-427e-82f1-caeeba236329
            Description:  Repository Update
                 Status:  Success
                Created:  February 26, 2026 12:54:57 CET
                Message:  /opt/dbi/odacli-dcs-19.30.0.0.0-260210-DB-19.30.0.0.zip

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Unzip bundle                             February 26, 2026 12:54:57 CET           February 26, 2026 12:55:50 CET           Success



odacli list-dbhomes
ID                                       Name                 DB Version           DB Edition Home Location                                            Status
---------------------------------------- -------------------- -------------------- ---------- -------------------------------------------------------- ----------
57c0dd7f-dcf4-4a38-9e79-4bf8c78e81bb     OraDB19000_home9     19.29.0.0.251021     EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_9     CONFIGURED

Let’s check if the patch can be applied:

odacli create-prepatchreport -d -i 57c0dd7f-dcf4-4a38-9e79-4bf8c78e81bb -v 19.30.0.0.0

sleep 600; odacli describe-prepatchreport -i 3522c79c-7444-44d7-9422-9d1daab161d2

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  3522c79c-7444-44d7-9422-9d1daab161d2
            Description:  Patch pre-checks for [DB, RHPDB, ORACHKDB] to 19.30.0.0.0: DbHome is OraDB19000_home9
                 Status:  FAILED
                Created:  February 26, 2026 12:56:59 PM CET
                 Result:  One or more pre-checks failed for [ORACHK, DB]

Node Name
---------------
dbioda01

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__DB__
Validate data corruption in     Failed    DCS-10315 - Patch described in My
patching                                  Oracle Support Note KB867473 must be
                                          applied.
Validate DB Home ID             Success   Validated DB Home ID:
                                          57c0dd7f-dcf4-4a38-9e79-4bf8c78e81bb
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate minimum agent version  Success   Validated minimum agent version
Is GI upgraded                  Success   Validated GI is upgraded
Validate available space for    Success   Validated free space required under
db                                        /u01
Validate there is usable        Success   Successfully validated Oracle Base
space under oracle base                   usable space
Validate glogin.sql file        Success   Successfully verified glogin.sql
                                          won't break patching
Validate dbHomesOnACFS          Success   User has configured disk group for
configured                                Database homes on ACFS
Validate Oracle base            Success   Successfully validated Oracle Base
Is DB clone available           Success   Successfully validated clone file
                                          exists
Validate command execution      Success   Validated command execution

__RHPDB__
Evaluate DBHome patching with   Success   Successfully validated updating
RHP                                       dbhome with RHP.  and local patching
                                          is possible
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Failed    DCS-10702 - ORAchk validation failed:
                                          .
Validate command execution      Success   Validated command execution
Verify the Fast Recovery Area   Failed    AHF-2929: FRA space management
(FRA) has reclaimable space               problem file types are present
                                          without an RMAN backup completion
                                          within the last 7 days

I need to fix 2 problems. The first one is a bug that appeared in 19.29, let’s download and unzip the patch. Be careful because this patch is available for multiple versions: you will need the one for the version you’re currently using (19.29 in my case).

su - oracle
unzip -d /home/oracle /opt/dbi/p38854064_1929000DBRU_Linux-x86-64.zip
Archive:  /opt/dbi/p38854064_1929000DBRU_Linux-x86-64.zip
   creating: /home/oracle/38854064/
   creating: /home/oracle/38854064/files/
   creating: /home/oracle/38854064/files/lib/
   creating: /home/oracle/38854064/files/lib/libserver19.a/
  inflating: /home/oracle/38854064/files/lib/libserver19.a/kjfc.o
  inflating: /home/oracle/38854064/README.txt
   creating: /home/oracle/38854064/etc/
   creating: /home/oracle/38854064/etc/config/
  inflating: /home/oracle/38854064/etc/config/inventory.xml
  inflating: /home/oracle/38854064/etc/config/actions.xml
  inflating: /home/oracle/PatchSearch.xml

Let’s stop the database and apply this patch:

. oraenv <<< CPROD19
srvctl stop database -db CPROD19_S1

cd 38854064
$ORACLE_HOME/OPatch/opatch apply
Oracle Interim Patch Installer version 12.2.0.1.47
Copyright (c) 2026, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_9
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_9/oraInst.loc
OPatch version    : 12.2.0.1.47
OUI version       : 12.2.0.7.0
Log file location : /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_9/cfgtoollogs/opatch/opatch2026-02-26_14-29-22PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   38854064

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_9')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '38854064' to OH '/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_9'

Patching component oracle.rdbms, 19.0.0.0.0...
Patch 38854064 successfully applied.
Log file location: /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_9/cfgtoollogs/opatch/opatch2026-02-26_14-29-22PM_1.log

OPatch succeeded.

srvctl start database -db CPROD19_S1

Second problem is because my database doesn’t have a proper backup strategy, let’s then remove the useless archivelogs:

rman target /
delete force noprompt archivelog all;
exit;
exit

Now let’s retry the precheck:

odacli create-prepatchreport -d -i 57c0dd7f-dcf4-4a38-9e79-4bf8c78e81bb -v 19.30.0.0.0

odacli describe-prepatchreport -i b932011d-cfcf-402d-901e-5c7eac888f1f

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  b932011d-cfcf-402d-901e-5c7eac888f1f
            Description:  Patch pre-checks for [DB, RHPDB, ORACHKDB] to 19.30.0.0.0: DbHome is OraDB19000_home9
                 Status:  FAILED
                Created:  February 26, 2026 3:01:14 PM CET
                 Result:  One or more pre-checks failed for [ORACHK, DB]

Node Name
---------------
dbioda01

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__DB__
Validate data corruption in     Failed    DCS-10315 - Patch described in My
patching                                  Oracle Support Note KB867473 must be
                                          applied.
Validate DB Home ID             Success   Validated DB Home ID:
                                          57c0dd7f-dcf4-4a38-9e79-4bf8c78e81bb
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate minimum agent version  Success   Validated minimum agent version
Is GI upgraded                  Success   Validated GI is upgraded
Validate available space for    Success   Validated free space required under
db                                        /u01
Validate there is usable        Success   Successfully validated Oracle Base
space under oracle base                   usable space
Validate glogin.sql file        Success   Successfully verified glogin.sql
                                          won't break patching
Validate dbHomesOnACFS          Success   User has configured disk group for
configured                                Database homes on ACFS
Validate Oracle base            Success   Successfully validated Oracle Base
Is DB clone available           Success   Successfully validated clone file
                                          exists
Validate command execution      Success   Validated command execution

__RHPDB__
Evaluate DBHome patching with   Success   Successfully validated updating
RHP                                       dbhome with RHP.  and local patching
                                          is possible
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Failed    DCS-10702 - ORAchk validation failed:
                                          .
Validate command execution      Success   Validated command execution
Verify the Fast Recovery Area   Failed    AHF-2929: FRA space management
(FRA) has reclaimable space               problem file types are present
                                          without an RMAN backup completion
                                          within the last 7 days

This view doesn’t look updated, and the ODA documentation tells us that updating the DB home will need to be forced, let’s do that:

odacli update-dbhome -i 57c0dd7f-dcf4-4a38-9e79-4bf8c78e81bb -v 19.30.0.0.0 --force

sleep 600;  odacli describe-job -i "bd511055-7a35-45b4-b9f2-3a003c7ecb31"
Job details
----------------------------------------------------------------
                     ID:  bd511055-7a35-45b4-b9f2-3a003c7ecb31
            Description:  DB Home Patching to 19.30.0.0.0: Home ID is 57c0dd7f-dcf4-4a38-9e79-4bf8c78e81bb
                 Status:  Success
                Created:  February 26, 2026 15:08:46 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Creating wallet for DB Client            February 26, 2026 15:09:33 CET           February 26, 2026 15:09:33 CET           Success
Patch databases by RHP - [CPROD19]       February 26, 2026 15:09:33 CET           February 26, 2026 15:16:44 CET           Success
Updating database metadata               February 26, 2026 15:16:44 CET           February 26, 2026 15:16:44 CET           Success
Upgrade pwfile to 12.2                   February 26, 2026 15:16:44 CET           February 26, 2026 15:16:47 CET           Success
Set log_archive_dest for Database        February 26, 2026 15:16:47 CET           February 26, 2026 15:16:50 CET           Success
Populate PDB metadata                    February 26, 2026 15:16:51 CET           February 26, 2026 15:16:52 CET           Success
Generating and saving BOM                February 26, 2026 15:16:52 CET           February 26, 2026 15:17:33 CET           Success
TDE parameter update                     February 26, 2026 15:18:06 CET           February 26, 2026 15:18:06 CET           Success

Everything is now OK.

Let’s check the DB homes and databases:

odacli list-dbhomes
ID                                       Name                 DB Version           DB Edition Home Location                                            Status
---------------------------------------- -------------------- -------------------- ---------- -------------------------------------------------------- ----------
d3b5fa9c-ad85-46c3-b11a-cd264978b653     OraDB19000_home10    19.30.0.0.260120     EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_10    CONFIGURED
57c0dd7f-dcf4-4a38-9e79-4bf8c78e81bb     OraDB19000_home9     19.29.0.0.251021     EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_9     CONFIGURED

odacli list-databases
ID                                       DB Name    DB Type  DB Version           CDB     Class    Edition  Shape    Storage  Status       DB Home ID
---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
976a80f2-4653-469f-8cd4-ddc1a21aff51     CPROD19    SI       19.30.0.0.260120     true    OLTP     EE       odb8     ASM      CONFIGURED   d3b5fa9c-ad85-46c3-b11a-cd264978b653

Let’s now remove the old DB home. Note that DB homes are not protected by ODABR, I would recommend doing a backup before removing an old DB home:

tar czf /backup/`hostname -s`_dbhome_9.tgz /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_9

odacli delete-dbhome -i 57c0dd7f-dcf4-4a38-9e79-4bf8c78e81bb
sleep 40 ; odacli describe-job -i 4589a4d7-6986-4e16-818c-78d585f44443
Job details
----------------------------------------------------------------
                     ID:  4589a4d7-6986-4e16-818c-78d585f44443
            Description:  Database Home OraDB19000_home9 Deletion with ID 57c0dd7f-dcf4-4a38-9e79-4bf8c78e81bb
                 Status:  Success
                Created:  February 26, 2026 15:24:44 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Setting up SSH equivalence               February 26, 2026 15:24:44 CET           February 26, 2026 15:24:44 CET           Success
Setting up SSH equivalence               February 26, 2026 15:24:44 CET           February 26, 2026 15:24:44 CET           Success
Validate DB Home                         February 26, 2026 15:24:44 CET           February 26, 2026 15:24:44 CET           Success
57c0dd7f-dcf4-4a38-9e79-4bf8c78e81bb
for deletion
Deleting DB Home by RHP                  February 26, 2026 15:24:45 CET           February 26, 2026 15:25:21 CET           Success

Cleanse the old patches

Let’s remove the previous patch from the repository:

odacli cleanup-patchrepo -comp all -v 19.29.0.0.0

odacli describe-job -i "a9e29414-8f12-4b55-a6d4-9ad82e9a4c74"
Job details
----------------------------------------------------------------
                     ID:  a9e29414-8f12-4b55-a6d4-9ad82e9a4c74
            Description:  Cleanup patchrepos
                 Status:  Success
                Created:  February 26, 2026 15:29:46 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Cleanup Repository                       February 26, 2026 15:29:46 CET           February 26, 2026 15:29:47 CET           Success
Cleanup old ASR rpm                      February 26, 2026 15:29:47 CET           February 26, 2026 15:29:47 CET           Success

Old GI binaries are still using space in /u01, it’s better to remove them manually:

du -hs /u01/app/19.*
14G	/u01/app/19.29.0.0
14G	/u01/app/19.30.0.0

rm -rf /u01/app/19.29.0.0

I would recommend doing a reboot to check if everything run fine. But let’s first check the components:

odacli describe-component
System Version
--------------
19.30.0.0.0

System Node Name
----------------
dbioda01

Local System Version
--------------------
19.30.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                      19.30.0.0.0          up-to-date

GI                                       19.30.0.0.260120     up-to-date

DB {
     OraDB19000_home10                   19.30.0.0.260120     up-to-date
     [CPROD19]
}

DCSCONTROLLER                            19.30.0.0.0          up-to-date

DCSCLI                                   19.30.0.0.0          up-to-date

DCSAGENT                                 19.30.0.0.0          up-to-date

DCSADMIN                                 19.30.0.0.0          up-to-date

OS                                       8.10                 up-to-date

ILOM                                     5.1.5.29.r167438     up-to-date

BIOS                                     52170100             up-to-date

LOCAL CONTROLLER FIRMWARE {
     [c4]                                8000D9AB             up-to-date
}

SHARED CONTROLLER FIRMWARE {
     [c0, c1]                            VDV1RL06             up-to-date
}

LOCAL DISK FIRMWARE {
     [c2d0, c2d1]                        XC311132             up-to-date
}

HMP                                      2.4.10.1.600         up-to-date

reboot
...

ps -ef | grep pmon
grid      8292     1  0 15:37 ?        00:00:00 asm_pmon_+ASM1
grid     11539     1  0 15:37 ?        00:00:00 apx_pmon_+APX1
oracle   20494     1  0 15:38 ?        00:00:00 ora_pmon_CPROD19
root     23559 23363  0 15:39 pts/1    00:00:00 grep --color=auto pmon

Everything is fine.

Post-patching tasks

Dont’ forget these post-patching tasks:

remove the ODABR snapshots
add your additional RPMs
put back your profile scripts for grid and oracle users
check if monitoring still works

/opt/odabr/odabr infosnap

│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
 odabr - ODA node Backup Restore - Version: 2.0.2-06
 Copyright 2013, 2025, Oracle and/or its affiliates.
 --------------------------------------------------------
 RACPack, Cloud Innovation and Solution Engineering Team
│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│


LVM snap name         status                COW Size              Data%
-------------         ----------            ----------            ------
root_snap             active                30.00 GiB             6.08%
opt_snap              active                70.00 GiB             11.41%
u01_snap              active                60.00 GiB             25.30%

/opt/odabr/odabr delsnap

INFO: 2026-02-26 15:39:47: Please check the logfile '/opt/odabr/out/log/odabr_23962.log' for more details


│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
 odabr - ODA node Backup Restore - Version: 2.0.2-06
 Copyright 2013, 2025, Oracle and/or its affiliates.
 --------------------------------------------------------
 RACPack, Cloud Innovation and Solution Engineering Team
│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│

INFO: 2026-02-26 15:39:47: Removing LVM snapshots
INFO: 2026-02-26 15:39:47: ...removing LVM snapshot for 'opt'
SUCCESS: 2026-02-26 15:39:48: ...snapshot for 'opt' removed successfully
INFO: 2026-02-26 15:39:48: ...removing LVM snapshot for 'u01'
SUCCESS: 2026-02-26 15:39:48: ...snapshot for 'u01' removed successfully
INFO: 2026-02-26 15:39:48: ...removing LVM snapshot for 'root'
SUCCESS: 2026-02-26 15:39:48: ...snapshot for 'root' removed successfully
SUCCESS: 2026-02-26 15:39:48: LVM snapshots removed successfully

Patching the DB System

If you use DB Systems on your ODA, meaning that some of your databases are running in dedicated VMs, you will need to apply the patch inside each DB System. If you’re using 26ai, you first need to register the new clones in the repository before connecting to your DB System:

odacli update-repository -f /opt/dbi/odacli-dcs-23.26.1.0.0-260211-GI-23.26.1.0.zip
sleep 30 ; odacli describe-job -i 8612ef6a-7df4-419d-8d05-176e11126f48

Job details
----------------------------------------------------------------
                     ID:  8612ef6a-7df4-419d-8d05-176e11126f48
            Description:  Repository Update
                 Status:  Success
                Created:  February 26, 2026 15:44:00 CET
                Message:  /opt/dbi/odacli-dcs-23.26.1.0.0-260211-GI-23.26.1.0.zip

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Unzip bundle                             February 26, 2026 15:44:00 CET           February 26, 2026 15:44:13 CET           Success


odacli update-repository -f /opt/dbi/odacli-dcs-23.26.1.0.0-260211-DB-23.26.1.0.zip
sleep 30 ; odacli describe-job -i 9dd624f2-9048-4897-b63b-400b955c803c

Job details
----------------------------------------------------------------
                     ID:  9dd624f2-9048-4897-b63b-400b955c803c
            Description:  Repository Update
                 Status:  Success
                Created:  February 26, 2026 15:45:12 CET
                Message:  /opt/dbi/odacli-dcs-23.26.1.0.0-260211-DB-23.26.1.0.zip

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Unzip bundle                             February 26, 2026 15:45:12 CET           February 26, 2026 15:45:40 CET           Success

odacli update-repository -f /opt/dbi/oda-sm-23.26.1.0.0-260211-server.zip
sleep 20 ; odacli describe-job -i 6a24023f-de2e-4481-b9c4-d8511d54be48

Job details
----------------------------------------------------------------
                     ID:  6a24023f-de2e-4481-b9c4-d8511d54be48
            Description:  Repository Update
                 Status:  Success
                Created:  February 26, 2026 15:59:07 CET
                Message:  /opt/dbi/oda-sm-23.26.1.0.0-260211-server.zip

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Unzip bundle                             February 26, 2026 15:59:08 CET           February 26, 2026 15:59:30 CET           Success

odacli list-availablepatches
-------------------- ------------------------- ------------------------- ------------------------------
ODA Release Version  Supported DB Versions     Available DB Versions     Supported Platforms
-------------------- ------------------------- ------------------------- ------------------------------
19.30.0.0.0          23.26.1.0.0               23.26.1.0.0               DB System
                     21.8.0.0.221018           Clone not available       DB System
                     19.30.0.0.260120          19.30.0.0.260120          DB System, Bare Metal

Applying the patch is done the same way you’ve done it on bare metal, but here you need to specify the 23.26.1.0.0 version:

ssh dbs-04-tst

odacli update-dcsadmin -v 23.26.1.0.0

sleep 60 ; odacli describe-job -i 4b83ab57-ccb1-4f9f-8c70-572b45ada49b

Job details
----------------------------------------------------------------
                     ID:  4b83ab57-ccb1-4f9f-8c70-572b45ada49b
            Description:  DcsAdmin patching to 23.26.1.0.0
                 Status:  Success
                Created:  March 05, 2026 10:07:44 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Dcs-admin upgrade                        March 05, 2026 10:07:45 CET              March 05, 2026 10:07:59 CET              Success
Ping DCS Admin                           March 05, 2026 10:07:59 CET              March 05, 2026 10:09:07 CET              Success

sleep 30 ;  odacli update-dcscomponents -v 23.26.1.0.0
{
  "jobId" : "cb674b3e-d6eb-4351-be39-0f19b8c56f9d",
  "status" : "Success",
  "message" : "Update-dcscomponents is successful on all the node(s): DCS-Agent shutdown is successful. MySQL upgrade is successful. Metadata schema update is done. Script '/opt/oracle/dcs/log/jobfiles/cb674b3e-d6eb-4351-be39-0f19b8c56f9d/apply_metadata_change.sh' ran successfully. dcsagent RPM upgrade is successful. dcscli RPM upgrade is successful. dcscontroller RPM upgrade is successful. ahf RPM upgrade is successful.  Successfully reset the Keystore password. HAMI RPM is already updated.  Removed old Libs Successfully ran setupAgentAuth.sh ",
  "reports" : null,
  "createTimestamp" : "March 05, 2026 10:10:13 AM CET",
  "description" : "Update-dcscomponents job completed and is not part of Agent job list",
  "updatedTime" : "March 05, 2026 10:18:38 AM CET",
  "jobType" : null,
  "externalRequestId" : null,
  "action" : null
}

odacli describe-admin-job -i cb674b3e-d6eb-4351-be39-0f19b8c56f9d
odacli: 'describe-admin-job' is not an odacli command.
usage: odacli	[-h/--help]
		 [-h/--help]
		 [-h/--help]
		 [-h/--help]
		 []

Note that there is no describe-admin-job feature on DB Systems.

odacli create-prepatchreport -sc -v 23.26.1.0.0

sleep 20 ; odacli describe-prepatchreport -i 1b104d06-bc0c-45b8-ab25-b5b6a102a857

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  1b104d06-bc0c-45b8-ab25-b5b6a102a857
            Description:  Patch pre-checks for [OS, ORACHKSERVER, SERVER] to 23.26.1.0.0
                 Status:  SUCCESS
                Created:  March 05, 2026 10:59:34 CET
                 Result:  All pre-checks succeeded

Node Name
---------------
dbs-04-tst

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 23.26.1.0.0.
Is patch location available     Success   Patch location is available.
Verify All OS patches           Success   No dependencies found for RPMs being
                                          removed, updated and installed. Check
                                          /opt/oracle/dcs/log/jobfiles/
                                          dnfdryrunout_2026-03-05_10-59-
                                          50.0718_832.log file for more details
Validate there is usable        Success   Successfully validated
space under repo volume                   /opt/oracle/dcs/repo usable space
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate all KVM ACFS           Success   All KVM ACFS resources are running
resources are running
Validate DB System VM states    Success   All DB System VMs states are expected
Enable support for Multi-DB     Success   No need to convert the DB System
Validate DB System AFD state    Success   AFD is not configured
Validate there is usable        Success   Successfully validated
space under repo volume                   /opt/oracle/dcs/repo usable space
Validate command execution      Success   Validated command execution


odacli update-servercomponents -v 23.26.1.0.0

The DB System will reboot.

odacli describe-job -i 2b4da73a-7f64-48e0-af76-a1d687a0169f

Job details
----------------------------------------------------------------
                     ID:  2b4da73a-7f64-48e0-af76-a1d687a0169f
            Description:  Server Patching to 23.26.1.0.0
                 Status:  Success
                Created:  March 05, 2026 11:04:19 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Deactivate Unit[dnf-makecache.timer]     March 05, 2026 11:04:21 CET              March 05, 2026 11:04:21 CET              Success
Validating GI user metadata              March 05, 2026 11:04:21 CET              March 05, 2026 11:04:21 CET              Success
Deactivate Unit[kdump.service]           March 05, 2026 11:04:22 CET              March 05, 2026 11:04:23 CET              Success
Modify DBVM udev rules                   March 05, 2026 11:04:23 CET              March 05, 2026 11:04:34 CET              Success
Creating local repository                March 05, 2026 11:04:34 CET              March 05, 2026 11:04:37 CET              Success
OSPatchBaseRepo
Updating versionlock plugin              March 05, 2026 11:04:37 CET              March 05, 2026 11:04:41 CET              Success
Applying OS Patches                      March 05, 2026 11:04:41 CET              March 05, 2026 11:07:33 CET              Success
Creating local repository HMPPatchRepo   March 05, 2026 11:07:34 CET              March 05, 2026 11:07:34 CET              Success
Applying HMP Patches                     March 05, 2026 11:07:35 CET              March 05, 2026 11:07:38 CET              Success
Patch location validation                March 05, 2026 11:07:39 CET              March 05, 2026 11:07:39 CET              Success
Setting SELinux mode                     March 05, 2026 11:07:39 CET              March 05, 2026 11:07:39 CET              Success
Oda-hw-mgmt upgrade                      March 05, 2026 11:07:40 CET              March 05, 2026 11:08:08 CET              Success
Installing SQLcl software                March 05, 2026 11:08:08 CET              March 05, 2026 11:08:13 CET              Success
Cleanup JRE Home                         March 05, 2026 11:08:14 CET              March 05, 2026 11:08:14 CET              Success
Generating and saving BOM                March 05, 2026 11:08:17 CET              March 05, 2026 11:08:24 CET              Success
Update System full patch version         March 05, 2026 11:08:17 CET              March 05, 2026 11:08:17 CET              Success
Update System rebootless patch version   March 05, 2026 11:08:17 CET              March 05, 2026 11:08:17 CET              Success
PreRebootNode Actions                    March 05, 2026 11:08:24 CET              March 05, 2026 11:08:25 CET              Success
Reboot Node                              March 05, 2026 11:08:25 CET              March 05, 2026 11:09:59 CET              Success

odacli create-prepatchreport -gi -v 23.26.1.0.0

sleep 240 ; odacli describe-prepatchreport -i dd5d216b-d1bc-44cf-bcf8-381da0729469

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  dd5d216b-d1bc-44cf-bcf8-381da0729469
            Description:  Patch pre-checks for [RHPGI, GI] to 23.26.1.0.0
                 Status:  SUCCESS
                Created:  March 05, 2026 11:13:21 CET
                 Result:  All pre-checks succeeded

Node Name
---------------
dbs-04-tst

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__RHPGI__
Validate available space        Success   Validated free space under /u01
Evaluate GI patching            Success   Successfully validated GI patching
Validate there is usable        Success   Successfully validated
space under repo volume                   /opt/oracle/dcs/repo usable space
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Successfully validated minimum version
Validate there is usable        Success   Successfully validated
space under repo volume                   /opt/oracle/dcs/repo usable space
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 23.26.1.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate BM versions            Success   Validated BM server components
                                          versions
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate command execution      Success   Validated command execution


odacli update-gihome -v 23.26.1.0.0

sleep 600 ; odacli describe-job -i c93f84fc-5cb2-41bb-9f23-f7ce22b9f5de

Job details
----------------------------------------------------------------
                     ID:  c93f84fc-5cb2-41bb-9f23-f7ce22b9f5de
            Description:  Patch GI with RHP to 23.26.1.0.0
                 Status:  Success
                Created:  March 05, 2026 11:22:47 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Patch GI with RHP to 23.26.1.0.0         March 05, 2026 11:22:59 CET              March 05, 2026 11:27:54 CET              Success
Starting the clusterware                 March 05, 2026 11:22:59 CET              March 05, 2026 11:22:59 CET              Success
Creating GI home directories             March 05, 2026 11:23:01 CET              March 05, 2026 11:23:01 CET              Success
Extract GI clone                         March 05, 2026 11:23:01 CET              March 05, 2026 11:23:01 CET              Success
Provisioning Software Only GI with RHP   March 05, 2026 11:23:01 CET              March 05, 2026 11:23:01 CET              Success
Registering image                        March 05, 2026 11:23:01 CET              March 05, 2026 11:23:01 CET              Success
Registering image                        March 05, 2026 11:23:01 CET              March 05, 2026 11:23:01 CET              Success
Registering working copy                 March 05, 2026 11:23:01 CET              March 05, 2026 11:23:01 CET              Success
Patch GI with RHP                        March 05, 2026 11:23:47 CET              March 05, 2026 11:26:58 CET              Success
Set CRS ping target                      March 05, 2026 11:26:58 CET              March 05, 2026 11:26:59 CET              Success
Updating .bashrc                         March 05, 2026 11:26:59 CET              March 05, 2026 11:26:59 CET              Success
Updating GI home metadata                March 05, 2026 11:26:59 CET              March 05, 2026 11:27:00 CET              Success
Updating GI home version                 March 05, 2026 11:27:00 CET              March 05, 2026 11:27:04 CET              Success
Updating All DBHome version              March 05, 2026 11:27:04 CET              March 05, 2026 11:27:08 CET              Success
Patch DB System on BM                    March 05, 2026 11:27:48 CET              March 05, 2026 11:27:54 CET              Success
Starting the clusterware                 March 05, 2026 11:27:48 CET              March 05, 2026 11:27:48 CET              Success


odacli list-dbhomes
ID                                       Name                 DB Version           DB Edition Home Location                                            Status
---------------------------------------- -------------------- -------------------- ---------- -------------------------------------------------------- ----------
9116603b-3b5e-4e92-aa63-baad8ae1d6a8     OraDB23000_home1     23.26.0.0.0          EE         /u01/app/oracle/product/23.0.0.0/dbhome_1                CONFIGURED

odacli create-prepatchreport -d -i 9116603b-3b5e-4e92-aa63-baad8ae1d6a8 -v 23.26.1.0.0

sleep 600 ;  odacli describe-prepatchreport -i bb16e390-3dcb-4ea0-b8c5-0c22f38ba271

odacli describe-prepatchreport -i bb16e390-3dcb-4ea0-b8c5-0c22f38ba271

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  bb16e390-3dcb-4ea0-b8c5-0c22f38ba271
            Description:  Patch pre-checks for [DB, RHPDB, ORACHKDB] to 23.26.1.0.0: DbHome is OraDB23000_home1
                 Status:  FAILED
                Created:  March 05, 2026 11:59:29 CET
                 Result:  One or more pre-checks failed for [ORACHK]

Node Name
---------------
dbs-04-tst

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__DB__
Validate DB Home ID             Success   Validated DB Home ID:
                                          9116603b-3b5e-4e92-aa63-baad8ae1d6a8
Validate patching tag           Success   Validated patching tag: 23.26.1.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate minimum agent version  Success   Validated minimum agent version
Is GI upgraded                  Success   Validated GI is upgraded
Validate available space for    Success   Validated free space required under
db                                        /u01
Validate there is usable        Success   Successfully validated Oracle Base
space under oracle base                   usable space
Validate glogin.sql file        Success   Successfully verified glogin.sql
                                          won't break patching
Is DB clone available           Success   Successfully validated clone file
                                          exists
Validate command execution      Success   Validated command execution

__RHPDB__
Evaluate DBHome patching with   Success   Successfully validated updating
RHP                                       dbhome with RHP.  and local patching
                                          is possible
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Failed    DCS-10702 - ORAchk validation failed:
                                          .
Validate command execution      Success   Validated command execution
Verify the Fast Recovery Area   Failed    AHF-2929: FRA space management
(FRA) has reclaimable space               problem file types are present
                                          without an RMAN backup completion
                                          within the last 7 days

The failure is similar to the one I had when patching the bare metal DB home, but I can ignore this and update the DB home with the force option:

odacli update-dbhome -i 9116603b-3b5e-4e92-aa63-baad8ae1d6a8 -v 23.26.1.0.0 -f

sleep 1200 ; odacli describe-job -i 4fc89556-2f7c-4e5b-a12f-55e32d7e748a

Job details
----------------------------------------------------------------
                     ID:  4fc89556-2f7c-4e5b-a12f-55e32d7e748a
            Description:  DB Home Patching to 23.26.1.0.0: Home ID is 9116603b-3b5e-4e92-aa63-baad8ae1d6a8
                 Status:  Success
                Created:  March 05, 2026 13:36:42 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Creating wallet for DB Client            March 05, 2026 13:37:21 CET              March 05, 2026 13:37:21 CET              Success
Patch databases by RHP - [CTEST26]       March 05, 2026 13:37:21 CET              March 05, 2026 13:54:26 CET              Success
Updating database metadata               March 05, 2026 13:54:26 CET              March 05, 2026 13:54:27 CET              Success
Upgrade pwfile to 12.2                   March 05, 2026 13:54:27 CET              March 05, 2026 13:54:32 CET              Success
Set log_archive_dest for Database        March 05, 2026 13:54:32 CET              March 05, 2026 13:54:37 CET              Success
Populate PDB metadata                    March 05, 2026 13:54:38 CET              March 05, 2026 13:54:39 CET              Success
Generating and saving BOM                March 05, 2026 13:54:39 CET              March 05, 2026 13:55:08 CET              Success
TDE parameter update                     March 05, 2026 13:55:44 CET              March 05, 2026 13:55:44 CET              Success

odacli list-databases
ID                                       DB Name    DB Type  DB Version           CDB     Class    Edition  Shape    Storage  Status       DB Home ID
---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
276bf458-db09-4c9a-9cd9-a821e5274fb0     CTEST26    SI       23.26.1.0.0          true    OLTP     EE       odb2     ASM      CONFIGURED   9c51039d-ccba-4508-b879-a81b8c18d46a

odacli delete-dbhome -i 9116603b-3b5e-4e92-aa63-baad8ae1d6a8
sleep 100 ; odacli describe-job -i 0994b96e-e174-4776-8699-f179c1d89af0

Job details
----------------------------------------------------------------
                     ID:  0994b96e-e174-4776-8699-f179c1d89af0
            Description:  Database Home OraDB23000_home1 Deletion with ID 9116603b-3b5e-4e92-aa63-baad8ae1d6a8
                 Status:  Success
                Created:  March 05, 2026 13:58:36 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Setting up SSH equivalence               March 05, 2026 13:58:36 CET              March 05, 2026 13:58:36 CET              Success
Setting up SSH equivalence               March 05, 2026 13:58:36 CET              March 05, 2026 13:58:37 CET              Success
Validate DB Home                         March 05, 2026 13:58:36 CET              March 05, 2026 13:58:36 CET              Success
9116603b-3b5e-4e92-aa63-baad8ae1d6a8
for deletion
Deleting DB Home by RHP                  March 05, 2026 13:58:38 CET              March 05, 2026 13:59:15 CET              Success

odacli describe-component
System Version
--------------
23.26.1.0.0

System Node Name
----------------
dbs-04-tst

Local System Version
--------------------
23.26.1.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                      23.26.1.0.0          up-to-date

GI                                       23.26.1.0.0          up-to-date

DB {
     OraDB23000_home2                    23.26.1.0.0          up-to-date
     [CTEST26]
}

DCSCONTROLLER                            23.26.1.0.0          up-to-date

DCSCLI                                   23.26.1.0.0          up-to-date

DCSAGENT                                 23.26.1.0.0          up-to-date

DCSADMIN                                 23.26.1.0.0          up-to-date

OS                                       8.10                 up-to-date

Finally, let’s remove obsolete GI binaries:

du -hs /u01/app/23.26.*
3.9G	/u01/app/23.26.0.0
3.6G	/u01/app/23.26.1.0

rm -rf /u01/app/23.26.0.0/

Don’t forget to apply this procedure to the other DB Systems.

Conclusion

Applying this patch is OK, as soon as everything is clean and under control. When patching, only use the force option when you’re sure that you know what you’re doing. As always, patching an ODA with DB Systems can take quite a big amount of time, mainly depending on the number of DB Systems.

L’article How to patch your ODA to 19.30? est apparu en premier sur dbi Blog.

Migrate your database to 26ai on ODA

Jérôme Dubar — Mon, 12 Jan 2026 14:36:59 +0000

Introduction

Oracle 26ai is not yet available on Bare Metal ODA, but it will probably be in a few weeks. For now, 26ai is available as a DB System on Oracle Database Appliance, meaning that it’s already possible to test this new version. Let’s try to migrate a 19c Bare Metal PDB to a 26ai DB System using 3 different methods.

Prerequisites

26ai is only available with the minimum release 19.29, meaning on ODA X8-2 and newer models. You will need to patch your ODA to 19.29 first if you want to test 26ai. 26ai is only compatible with multitenant architecture, so another prerequisite is to migrate your non-container databases to PDBs. If you’re still using non-CDB architecture, and among these 3 methods, only the first one will work for you.

The recommended patching method as told by Oracle is using AutoUpgrade: please refer to this blog post if you want more details.

On my side, I will explore these 3 other possibilities:

Datapump Export/Import
PDB copy through a DB link
PDB unplug/plug

Download the needed files for 26ai DB System

These files are needed for creating a 26ai DB System:

36524660 => System image for 26ai DB Systems
36524627 => the GI clone needed for deploying newer 26ai GI version
36524642 => the DB clone for deploying new version 26ai

Be sure to choose the very latest 19.29 patch when downloading these files, download link from MOS will first propose older versions.

My 19c source database

My source database is a 19c container database with one PDB running on an ODA X8-2M. I want to migrate this PDB to 26ai:

su - oracle
 . oraenv <<< CPROD19
ORACLE_SID = [CPROD19] ? The Oracle base remains unchanged with value /u01/app/odaorabase/oracle
sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Thu Jan 8 09:53:43 2026
Version 19.29.0.0.0

Copyright (c) 1982, 2025, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.29.0.0.0

show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PRODDB01                       READ WRITE NO

exit

Provision a new 26ai DB System

It’s not yet possible to deploy a 26ai container database on Bare Metal, so let’s create a DB System for testing this migration. First unzip and register the VM template, GI 26ai and DB 26ai:

cd /opt/dbi

unzip -o p36524660_1929000_Linux-x86-64.zip
unzip -o p36524627_1929000_Linux-x86-64.zip
unzip -o p36524642_1929000_Linux-x86-64.zip

odacli update-repository -f /opt/dbi/odacli-dcs-23.26.0.0.0-251116-ODAVM-19.29.0.0.zip
odacli update-repository -f /opt/dbi/odacli-dcs-23.26.0.0.0-251116-GI-23.26.0.0.zip
odacli update-repository -f /opt/dbi/odacli-dcs-23.26.0.0.0-251116-DB-23.26.0.0.zip

sleep 30 ; odacli list-jobs | tail -n 4
d4700315-db8c-4522-af55-0fddd262bfe4     Repository Update                                                           2025-12-08 15:20:14 CET             Success
ba7d452f-e03d-46d0-a607-fd7c758cd1b1     Repository Update                                                           2025-12-08 15:20:59 CET             Success
bfc102d1-985b-4792-8054-03709aa8d949     Repository Update                                                           2025-12-08 15:21:20 CET             Success

odacli describe-dbsystem-image | grep 23.26
DBVM                  23.26.0.0.0           23.26.0.0.0
GI                    23.26.0.0.0           23.26.0.0.0
DB                    23.26.0.0.0           23.26.0.0.0

A CPU pool is needed for DB Systems, if you don’t have one. A 2-core CPU pool is enough:

odacli create-cpupool -c 2 -n cpupool2c -dbs

Now let’s create a json file, then create a new DB System based on this file:

cat /opt/dbi/create_dbs-04-tst-cdb.json
{
    "system": {
        "name": "dbs-04-tst",
        "shape": "dbs2",
        "systemPassword": "***********",
        "timeZone": "Europe/Zurich",
        "diskGroup": "DATA",
        "cpuPoolName": "cpupool2c",
        "enableRoleSeparation": true,
        "customRoleSeparation": {
            "groups": [
                {
                    "name": "oinstall",
                    "id": 1001,
                    "role": "oinstall"
                },
                {
                    "name": "dbaoper",
                    "id": 1002,
                    "role": "dbaoper"
                },
                {
                    "name": "dba",
                    "id": 1003,
                    "role": "dba"
                },
                {
                    "name": "asmadmin",
                    "id": 1004,
                    "role": "asmadmin"
                },
                {
                    "name": "asmoper",
                    "id": 1005,
                    "role": "asmoper"
                },
                {
                    "name": "asmdba",
                    "id": 1006,
                    "role": "asmdba"
                }
            ],
            "users": [
                {
                    "name": "grid",
                    "id": 1001,
                    "role": "gridUser"
                },
                {
                    "name": "oracle",
                    "id": 1000,
                    "role": "oracleUser"
                }
            ]
        }
    },
    "database": {
        "name": "CTEST26",
        "uniqueName": "CTEST26_S1",
        "domainName": "dbi-lab.ch",
        "adminPassword": "**********",
        "version": "23.26.0.0.0",
        "edition": "EE",
        "type": "SI",
        "dbClass": "OLTP",
        "shape": "odb2",
        "role": "PRIMARY",
        "enableDbConsole": false,
        "enableFlashStorage": false,
        "redundancy": null,
        "characterSet": {
            "characterSet": "AL32UTF8",
            "nlsCharacterset": "AL16UTF16",
            "dbTerritory": "AMERICA",
            "dbLanguage": "AMERICAN"
        },
        "rmanBackupPassword": null,
        "enableTDE": false,
        "isCdb": true,
        "pdbName": "pdb1",
        "pdbAdminUser": "pdbadmin",
        "tdePassword": null
    },
    "network": {
        "domainName": "dbi-lab.ch",
        "ntpServers": [
            "85.195.224.28"
        ],
        "dnsServers": [
            "10.16.0.250"
        ],
        "nodes": [
            {
                "name": "dbs-04-tst",
                "ipAddress": "10.16.0.247",
                "netmask": "255.255.255.0",
                "gateway": "10.16.0.1",
                "number": 0
            }
        ],
        "publicVNetwork": "pubnet"
    },
    "grid": {
        "language": "en"
    }
}

odacli create-dbsystem -p /opt/dbi/create_dbs-04-tst-cdb.json
Enter password for system "dbs-04-tst":
Retype password for system "dbs-04-tst":
Enter administrator password for DB "CTEST26":
Retype administrator password for DB "CTEST26":
...

35 minutes later, my new DB System is ready to use:

odacli describe-job -i c79ecbbf-dd0c-4b75-ba10-249c78b25f33

Job details
----------------------------------------------------------------
                     ID:  c79ecbbf-dd0c-4b75-ba10-249c78b25f33
            Description:  DB System dbs-04-tst creation
                 Status:  Success
                Created:  January 08, 2026 10:23:39 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Create DB System metadata                January 08, 2026 10:23:39 CET            January 08, 2026 10:23:41 CET            Success
Persist new DB System                    January 08, 2026 10:23:41 CET            January 08, 2026 10:23:41 CET            Success
Remove unused network interfaces         January 08, 2026 10:23:41 CET            January 08, 2026 10:23:41 CET            Success
Validate DB System creation              January 08, 2026 10:23:41 CET            January 08, 2026 10:23:46 CET            Success
Create the infra quota group             January 08, 2026 10:23:46 CET            January 08, 2026 10:23:50 CET            Success
Setup DB System environment              January 08, 2026 10:23:46 CET            January 08, 2026 10:23:46 CET            Success
Create DB System ASM volume              January 08, 2026 10:23:50 CET            January 08, 2026 10:24:01 CET            Success
Create DB System ACFS filesystem         January 08, 2026 10:24:01 CET            January 08, 2026 10:24:13 CET            Success
Create DB System ACFS mount point        January 08, 2026 10:24:01 CET            January 08, 2026 10:24:01 CET            Success
Create DB System VM ACFS snapshots	 January 08, 2026 10:24:13 CET            January 08, 2026 10:24:28 CET            Success
Setup ASM client cluster config          January 08, 2026 10:24:28 CET            January 08, 2026 10:24:30 CET            Success
Create temporary SSH key pair            January 08, 2026 10:24:30 CET            January 08, 2026 10:24:31 CET            Success
Create BM certificates in ACFS location  January 08, 2026 10:24:31 CET            January 08, 2026 10:24:32 CET            Success
Create DB System cloud-init config	 January 08, 2026 10:24:32 CET            January 08, 2026 10:24:32 CET            Success
Provision DB System VM(s)                January 08, 2026 10:24:32 CET            January 08, 2026 10:24:35 CET            Success
Create u01 vDisk for DB System           January 08, 2026 10:24:35 CET            January 08, 2026 10:24:36 CET            Success
Add DB System to Clusterware             January 08, 2026 10:24:36 CET            January 08, 2026 10:24:37 CET            Success
Attach disks to DB System                January 08, 2026 10:24:36 CET            January 08, 2026 10:24:36 CET            Success
Attach u01 vDisk to DB System            January 08, 2026 10:24:36 CET            January 08, 2026 10:24:36 CET            Success
Add extra PCI slots                      January 08, 2026 10:24:37 CET            January 08, 2026 10:24:37 CET            Success
Create Network Filters                   January 08, 2026 10:24:37 CET            January 08, 2026 10:24:37 CET            Success
Export clones repository                 January 08, 2026 10:24:37 CET            January 08, 2026 10:24:37 CET            Success
Start DB System                          January 08, 2026 10:24:37 CET            January 08, 2026 10:24:40 CET            Success
Wait DB System VM first boot             January 08, 2026 10:24:40 CET            January 08, 2026 10:26:49 CET            Success
Setup Mutual TLS (mTLS)                  January 08, 2026 10:26:49 CET            January 08, 2026 10:26:51 CET            Success
Cleanup temporary SSH key pair           January 08, 2026 10:26:51 CET            January 08, 2026 10:26:52 CET            Success
Copy ASM client cluster config           January 08, 2026 10:26:51 CET            January 08, 2026 10:26:51 CET            Success
Delete Mutual TLS (mTLS) Artifacts	 January 08, 2026 10:26:51 CET            January 08, 2026 10:26:51 CET            Success
Wait DB System VM DCS Agent readiness    January 08, 2026 10:26:52 CET            January 08, 2026 10:27:12 CET            Success
Add Network Filters                      January 08, 2026 10:27:12 CET            January 08, 2026 10:27:14 CET            Success
Install DB System                        January 08, 2026 10:27:14 CET            January 08, 2026 10:53:57 CET            Success
Wait DB System VM DCS Agent readiness    January 08, 2026 10:53:57 CET            January 08, 2026 10:55:58 CET            Success
Set DB System as configured              January 08, 2026 10:55:58 CET            January 08, 2026 10:55:58 CET            Success

odacli list-dbsystems

Name                  Shape       GI version          DB info                         Status                  Created                   Updated
--------------------  ----------  ------------------  ------------------------------  ----------------------  ------------------------  ------------------------
dbs-04-tst            dbs2        23.26.0.0.0         23.26(CONFIGURED=1)             CONFIGURED              2026-01-08 10:23:39 CET   2026-01-08 10:55:58 CET

Migration using datapump

Let’s consider a classic migration using datapump. This tool has been the official one for migration for nearly 2 decades now.

Let’s first create a new PDB inside the container database of the new DB System:

ssh root@dbs-04-tst
su - oracle
. oraenv <<< CTEST26
sqlplus / as sysdba
show pdbs
    CON_ID CON_NAME			  OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
	 2 PDB$SEED			  READ ONLY  NO
	 3 PDB1 			  READ WRITE NO

alter pluggable database PDB1 close immediate;
Pluggable database altered.

drop pluggable database PDB1 including datafiles;
Pluggable database dropped.

create pluggable database MIG26DTPUMP admin user pdbadmin identified by "*********";
Pluggable database created.

alter pluggable database MIG26DTPUMP open;
Pluggable database altered.

alter pluggable database MIG26DTPUMP save state;
Pluggable database altered.

show pdbs
    CON_ID CON_NAME			  OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
	 2 PDB$SEED			  READ ONLY  NO
	 4 MIG26DTPUMP			  READ WRITE NO

alter session set container=MIG26DTPUMP;

create tablespace users datafile size 100M autoextend on maxsize 10G;
Tablespace created.

exit

On source database, non-CDB or PDB it doesn’t matter, export data using datapump:

su - oracle
. oraenv <<< CPROD19
export ORACLE_PDB_SID=PRODDB01
sqlplus / as sysdba
create or replace directory mig as '/backup/dumps/';
col current_scn for 999999999999
select current_scn from v$database;
    CURRENT_SCN
---------------
        6292616
exit
expdp system directory=MIG full=y dumpfile=PRODDB01_20260112.dmp logfile=exp_PRODDB01_20260112.log flashback_scn=6292616

Export: Release 19.0.0.0.0 - Production on Mon Jan 12 10:36:52 2026
Version 19.29.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.
Password:

Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Starting "SYSTEM"."SYS_EXPORT_FULL_01":  system/******** directory=MIG full=y dumpfile=PRODDB01_20260112.dmp logfile=exp_PRODDB01_20260112.log flashback_scn=6292616
Processing object type DATABASE_EXPORT/EARLY_OPTIONS/VIEWS_AS_TABLES/TABLE_DATA
Processing object type DATABASE_EXPORT/NORMAL_OPTIONS/TABLE_DATA
Processing object type DATABASE_EXPORT/NORMAL_OPTIONS/VIEWS_AS_TABLES/TABLE_DATA
Processing object type DATABASE_EXPORT/SCHEMA/TABLE/TABLE_DATA
Processing object type DATABASE_EXPORT/SCHEMA/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
Processing object type DATABASE_EXPORT/SCHEMA/TABLE/STATISTICS/TABLE_STATISTICS
Processing object type DATABASE_EXPORT/STATISTICS/MARKER
Processing object type DATABASE_EXPORT/PRE_SYSTEM_IMPCALLOUT/MARKER
Processing object type DATABASE_EXPORT/PRE_INSTANCE_IMPCALLOUT/MARKER
Processing object type DATABASE_EXPORT/TABLESPACE
...

. . exported "WMSYS"."WM$METADATA_MAP"                   11.16 KB       0 rows
. . exported "MDSYS"."RDF_PARAM$"                        6.515 KB       3 rows
. . exported "SYS"."AUDTAB$TBS$FOR_EXPORT"               5.960 KB       2 rows
. . exported "SYS"."DBA_SENSITIVE_DATA"                  8.070 KB       0 rows
. . exported "SYS"."DBA_TSDP_POLICY_PROTECTION"          7.656 KB       0 rows
. . exported "SYS"."NACL$_ACE_EXP"                       9.812 KB       0 rows
. . exported "SYS"."NACL$_HOST_EXP"                      6.921 KB       1 rows
. . exported "SYS"."NACL$_WALLET_EXP"                    5.929 KB       0 rows
. . exported "DBI"."CONSULTANTS"                         5.937 KB       1 rows
Master table "SYSTEM"."SYS_EXPORT_FULL_01" successfully loaded/unloaded
******************************************************************************
Dump file set for SYSTEM.SYS_EXPORT_FULL_01 is:
  /backup/dumps/PRODDB01_20260112.dmp
Job "SYSTEM"."SYS_EXPORT_FULL_01" successfully completed at Mon Jan 12 10:38:52 2026 elapsed 0 00:01:54


ls -lrth /backup/dumps/*20260112*
-rw-r-----. 1 oracle asmadmin  34M Jan 12 10:38 /backup/dumps/PRODDB01_20260112.dmp
-rw-r--r--. 1 oracle asmadmin 9.8K Jan 12 10:38 /backup/dumps/exp_PRODDB01_20260112.log

On target database, let’s do the import:

ssh root@dbs-04-tst
su - oracle
. oraenv <<< CTEST26
export ORACLE_PDB_SID=MIG26DTPUMP
sqlplus / as sysdba
create or replace directory mig as '/backup/dumps/';
exit

vi /backup/dumps/imp_PRODDB01.par
directory=MIG
schemas=dbi
dumpfile=PRODDB01_20260112.dmp
logfile=imp_PRODDB01_to_MIG26DTPUMP_20260112.log
exclude=statistics

impdp system parfile=/backup/dumps/imp_PRODDB01.par
Import: Release 23.26.0.0.0 - for Oracle Cloud and Engineered Systems on Mon Jan 12 10:51:39 2026
Version 23.26.0.0.0

Copyright (c) 1982, 2025, Oracle and/or its affiliates.  All rights reserved.
Password:

Connected to: Oracle AI Database 26ai Enterprise Edition Release 23.26.0.0.0 - for Oracle Cloud and Engineered Systems
Master table "SYSTEM"."SYS_IMPORT_SCHEMA_01" successfully loaded/unloaded
import done in AL32UTF8 character set and AL16UTF16 NCHAR character set
export done in WE8MSWIN1252 character set and AL16UTF16 NCHAR character set
Warning: possible data loss in character set conversions
Starting "SYSTEM"."SYS_IMPORT_SCHEMA_01":  system/******** parfile=/backup/dumps/imp_PRODDB01.par
Processing object type DATABASE_EXPORT/SCHEMA/USER
Processing object type DATABASE_EXPORT/SCHEMA/GRANT/SYSTEM_GRANT
Processing object type DATABASE_EXPORT/SCHEMA/ROLE_GRANT
Processing object type DATABASE_EXPORT/SCHEMA/DEFAULT_ROLE
Processing object type DATABASE_EXPORT/SCHEMA/PROCACT_SCHEMA/LOGREP
Processing object type DATABASE_EXPORT/SCHEMA/TABLE/TABLE
Processing object type DATABASE_EXPORT/SCHEMA/TABLE/TABLE_DATA
. . imported "DBI"."CONSULTANTS"                           5.9 KB       1 rows
Job "SYSTEM"."SYS_IMPORT_SCHEMA_01" successfully completed at Mon Jan 12 10:51:54 2026 elapsed 0 00:00:08

Let’s now compute statistics for my schema:

sqlplus / as sysdba
exec dbms_stats.gather_schema_stats(ownname => 'DBI');
PL/SQL procedure successfully completed.
exit;

Let’s do a quick check on data:

sqlplus / as sysdba
select start_date from dbi.consultants where name like 'DUB%';

START_DAT
---------
04-SEP-17

exit

That’s fine.

Pros & cons

+ Same procedure as previous migrations
+ Reorganize data and indexes
+ Keep the source database untouched
– Rather slow when database is big

Migration using PDB copy through DB link

If the source database is a PDB, it can be copied through a DB link, this is a very convenient way of migrating a database.

First, it’s weird but the PDB administrator of the source DB must have the CREATE PLUGGABLE DATABASE privilege:

su - oracle
. oraenv <<< CPROD19
sqlplus / as sysdba
alter session set container=PRODDB01;
grant create pluggable database to pdbadmin;
exit

On the DB System, let’s create a new TNS entry for the 19c PDB:

ssh dbs-04-tst
su - oracle
. oraenv <<< CTEST26
vi $ORACLE_HOME/network/admin/tnsnames.ora

...
PRODDB01 =
  (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = dbioda01.dbi-lab.ch)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = proddb01.dbi-lab.ch)
    )
  )

On the target CDB, create a new database link to the source database:

CREATE DATABASE LINK PRODDB01 CONNECT TO PDBADMIN IDENTIFIED BY "*********"  USING 'PRODDB01';

select count(*) from dual@PRODDB01;
  COUNT(*)
----------
         1

Now let’s copy the source PDB under a new name on the 26ai container:

create pluggable database MIG26DBLINK from PRODDB01@PRODDB01;
Pluggable database created.

alter pluggable database MIG26DBLINK open;

Opening this PDB will take some minutes because it automatically triggers the upgrade from 19c to 26ai:

tail -n 15 --f /u01/app/oracle/diag/rdbms/ctest26_s1/CTEST26/trace/alert_CTEST26.log 
...
MIG26DBLINK(3):Starting Upgrade on PDB Open
2026-01-12T10:57:49.606651+01:00
MIG26DBLINK(3):alter pluggable database application APP$CDB$CATALOG begin install '19.0.0.0.0' on error capture
MIG26DBLINK(3):Completed: alter pluggable database application APP$CDB$CATALOG begin install '19.0.0.0.0' on error capture
MIG26DBLINK(3):alter pluggable database application APP$CDB$CATALOG end install '19.0.0.0.0'
MIG26DBLINK(3):Completed: alter pluggable database application APP$CDB$CATALOG end install '19.0.0.0.0'
MIG26DBLINK(3):alter pluggable database application APP$CDB$CATALOG begin upgrade
  '19.0.0.0.0' to '23.0.0.0.0.partial' on error capture
MIG26DBLINK(3):Completed: alter pluggable database application APP$CDB$CATALOG begin upgrade
  '19.0.0.0.0' to '23.0.0.0.0.partial' on error capture
...
MIG26DBLINK(3):SERVER COMPONENT id=CATJAVA: status=UPGRADED, version=23.26.0.0.0, timestamp=2026-01-12 11:09:54 Container=MIG26DBLINK Id=3
2026-01-12T11:09:58.200636+01:00
MIG26DBLINK(3):Resize operation completed for file# 19, fname +DATA/CTEST26_S1/482EE16D86417106E063F700240A6555/DATAFILE/undotbs1.294.1222340161, old size 450560K, new size 455680K
2026-01-12T11:09:59.496448+01:00
MIG26DBLINK(3):Resize operation completed for file# 19, fname +DATA/CTEST26_S1/482EE16D86417106E063F700240A6555/DATAFILE/undotbs1.294.1222340161, old size 455680K, new size 460800K
2026-01-12T11:10:26.701609+01:00
MIG26DBLINK(3):oracle :
MIG26DBLINK(3):SERVER COMPONENT id=XDB: status=UPGRADED, version=23.26.0.0.0, timestamp=2026-01-12 11:10:26 Container=MIG26DBLINK Id=3
2026-01-12T11:10:36.378304+01:00
MIG26DBLINK(3):Resize operation completed for file# 19, fname +DATA/CTEST26_S1/482EE16D86417106E063F700240A6555/DATAFILE/undotbs1.294.1222340161, old size 460800K, new size 465920K
2026-01-12T11:10:45.077998+01:00
MIG26DBLINK(3):Resize operation completed for file# 19, fname +DATA/CTEST26_S1/482EE16D86417106E063F700240A6555/DATAFILE/undotbs1.294.1222340161, old size 465920K, new size 471040K
...
MIG26DBLINK(3):***************************************************************
MIG26DBLINK(3):WARNING: Pluggable Database MIG26DBLINK with pdb id - 3 is
MIG26DBLINK(3):         altered with errors or warnings. Please look into
MIG26DBLINK(3):         PDB_PLUG_IN_VIOLATIONS view for more details.
MIG26DBLINK(3):***************************************************************
MIG26DBLINK(3):--ATTENTION--
MIG26DBLINK(3):Errors reported while opening PDB (ContainerId: 3) and have been recorded in pdb_alert$ table.
MIG26DBLINK(3) Error Violation: SQL Patch, Cause: '23.26.0.0.0 Release_Update 2509270520' is installed in the CDB but no release updates are installed in the PDB, Action: Call datapatch to install in the PDB or the CDB
2026-01-12T11:13:31.632629+01:00
MIG26DBLINK(3):Opening pdb with no Resource Manager plan active
Violations: Type: 1, Count: 1
Violations: Type: 2, Count: 2
Completed: Pluggable database MIG26DBLINK opened read write
Completed: alter pluggable database MIG26DBLINK open

After a couple of minutes, the PDB is opened with errors:

Warning: PDB altered with errors.
exit

Let’s apply datapatch on this very PDB:

$ORACLE_HOME/OPatch/datapatch -pdbs MIG26DBLINK
SQL Patching tool version 23.26.0.0.0 Lim on Mon Jan 12 11:14:55 2026
Copyright (c) 2012, 2025, Oracle.  All rights reserved.

Log file for this invocation: /u01/app/oracle/product/23.0.0.0/dbhome_1/cfgtoollogs/sqlpatch/sqlpatch_sid_CTEST26_ts_2026_01_12_11_14_55_pid_5074/sqlpatch_invocation.log

Connecting to database...OK
Gathering database info...done

Note:  Datapatch will only apply or rollback SQL fixes for PDBs
       that are in an open state, no patches will be applied to closed PDBs.
       Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
       (Doc ID 1585822.1)

Bootstrapping registry and package to current versions...done
Determining current state...done

Current state of interim SQL patches:
  No interim patches found

Current state of release update SQL patches:
  Binary registry:
    23.26.0.0.0 Release_Update 250927052030: Installed
  PDB MIG26DBLINK:
    Applied 23.26.0.0.0 Release_Update 250927052030 successfully

Adding patches to installation queue and performing prereq checks...done
Installation queue:
  For the following PDBs: MIG26DBLINK
    No interim patches need to be rolled back
    No release update patches need to be installed
    No interim patches need to be applied

Bypass install queue:
  For the following PDBs: MIG26DBLINK
    No interim rollbacks will bypass install
    Patch 38404116 (Database Release Update : 23.26.0.0.0 (38404116) Gold Image): will bypass install
      Apply from 23.26.0.0.0 Release_Update 250927052030 to 23.26.0.0.0 Release_Update 250927052030
    No interim applys will bypass install


Installation queue after removing bypass entries...
Installation queue:
  For the following PDBs: MIG26DBLINK
    No interim patches need to be rolled back
    No release update patches need to be installed
    No interim patches need to be applied


Processing bypass install queue:
  Patch 38404116 apply (pdb MIG26DBLINK): SUCCESS (bypass_install)

SQL Patching tool complete on Mon Jan 12 11:15:03 2026

Everything is fine, now let’s close and open this PDB in normal mode:

sqlplus / as sysdba
show pdbs

    CON_ID CON_NAME			  OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
	 2 PDB$SEED			  READ ONLY  NO
	 3 MIG26DBLINK			  READ WRITE YES
	 4 MIG26DTPUMP			  READ WRITE NO

alter pluggable database MIG26DBLINK close immediate;
alter pluggable database MIG26DBLINK open;
alter pluggable database MIG26DBLINK save state;

show pdbs

    CON_ID CON_NAME			  OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
	 2 PDB$SEED			  READ ONLY  NO
	 3 MIG26DBLINK			  READ WRITE NO
	 4 MIG26DTPUMP			  READ WRITE NO

Let’s do a quick data check:

sqlplus / as sysdba
alter session set container=MIG26DBLINK;
select start_date from dbi.consultants where name like 'DUB%';
START_DAT
---------
04-SEP-17

exit

Database is ready to use.

Pros & cons

+ Maybe the easiest way of migrating a PDB
+ Keep the source database untouched
+ Rather fast
+ No downtime of the source database
– Need to apply datapatch

Migration using unplug/plug features

If the source database is a PDB, it can be unplugged and plugged back to another container. This is how multitenant architecture was presented when it first came to 12cR1.

Let’s unplug the source PDB:

su - oracle
. oraenv <<< CPROD19
sqlplus / as sysdba
alter pluggable database PRODDB01 close immediate;
Pluggable database altered.

alter pluggable database PRODDB01 unplug into '/backup/dumps/PRODDB01.xml';
Pluggable database altered.

exit

Now let’s plug this PDB to the new 26ai container:

ssh root@dbs-04-tst
su - oracle
. oraenv <<< CTEST26

sqlplus / as sysdba

show pdbs

    CON_ID CON_NAME			  OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
	 2 PDB$SEED			  READ ONLY  NO
	 3 MIG26DBLINK			  READ WRITE NO
	 4 MIG26DTPUMP			  READ WRITE NO

create pluggable database MIG26REPLUG using '/backup/dumps/PRODDB01.xml' COPY;
create pluggable database MIG26REPLUG using '/backup/dumps/PRODDB01.xml' COPY
*
ERROR at line 1:
ORA-19505: failed to identify file
"+DATA/CPROD19_S1/450AA2873245B6F5E063F100240AC35A/DATAFILE/system.302.122233608
1"
ORA-17503: ksfdopn:2 Failed to open file
+DATA/CPROD19_S1/450AA2873245B6F5E063F100240AC35A/DATAFILE/system.302.1222336081
ORA-15260: permission denied on ASM disk group
Help: https://docs.oracle.com/error-help/db/ora-19505/

Files from the source database are not readable by this DB System. Let’s change the permission on these files from the Bare Metal ASM instance:

exit
exit
exit
su - grid

asmcmd ls -lt +DATA/CPROD19_S1/450AA2873245B6F5E063F100240AC35A/DATAFILE/* --permission
Type      Redund  Striped  Time             Sys  User                                     Group  Permission  Name
DATAFILE  MIRROR  COARSE   JAN 12 11:00:00  Y    oracle@e7d9d36f6b157fdabf89026bcf48d757          rw-------  USERS.297.1222336081
DATAFILE  MIRROR  COARSE   JAN 12 11:00:00  Y    oracle@e7d9d36f6b157fdabf89026bcf48d757          rw-------  UNDOTBS1.306.1222336081
DATAFILE  MIRROR  COARSE   JAN 12 11:00:00  Y    oracle@e7d9d36f6b157fdabf89026bcf48d757          rw-------  SYSTEM.302.1222336081
DATAFILE  MIRROR  COARSE   JAN 12 11:00:00  Y    oracle@e7d9d36f6b157fdabf89026bcf48d757          rw-------  SYSAUX.305.1222336081

asmcmd ls -lt +DATA/CTEST26_S1/DATAFILE/* --permission
Type      Redund  Striped  Time             Sys  User                                     Group  Permission  Name
DATAFILE  MIRROR  COARSE   JAN 12 10:00:00  Y    oracle@b94b488f82b1cf5cbf58f54162aafac0          rw-------  USERS.299.1222339507
DATAFILE  MIRROR  COARSE   JAN 12 10:00:00  Y    oracle@b94b488f82b1cf5cbf58f54162aafac0          rw-------  UNDOTBS1.693.1222338993
DATAFILE  MIRROR  COARSE   JAN 12 10:00:00  Y    oracle@b94b488f82b1cf5cbf58f54162aafac0          rw-------  SYSTEM.906.1222338951
DATAFILE  MIRROR  COARSE   JAN 12 10:00:00  Y    oracle@b94b488f82b1cf5cbf58f54162aafac0          rw-------  SYSAUX.832.1222338987

asmcmd chmod a+r +DATA/CPROD19_S1/450AA2873245B6F5E063F100240AC35A/DATAFILE/*
asmcmd ls -lt +DATA/CPROD19_S1/450AA2873245B6F5E063F100240AC35A/DATAFILE/* --permission
Type      Redund  Striped  Time             Sys  User                                     Group  Permission  Name
DATAFILE  MIRROR  COARSE   JAN 12 11:00:00  Y    oracle@e7d9d36f6b157fdabf89026bcf48d757          rw-r--r--  USERS.297.1222342153
DATAFILE  MIRROR  COARSE   JAN 12 11:00:00  Y    oracle@e7d9d36f6b157fdabf89026bcf48d757          rw-r--r--  UNDOTBS1.301.1222341995
DATAFILE  MIRROR  COARSE   JAN 12 11:00:00  Y    oracle@e7d9d36f6b157fdabf89026bcf48d757          rw-r--r--  SYSTEM.305.1222341995
DATAFILE  MIRROR  COARSE   JAN 12 11:00:00  Y    oracle@e7d9d36f6b157fdabf89026bcf48d757          rw-r--r--  SYSAUX.302.1222341995

exit

ssh root@dbs-04-tst
su - oracle
. oraenv <<< CTEST26

sqlplus / as sysdba

create pluggable database MIG26REPLUG using '/backup/dumps/PRODDB01.xml' COPY;
Pluggable database created.

show pdbs

    CON_ID CON_NAME			  OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
	 2 PDB$SEED			  READ ONLY  NO
	 3 MIG26DBLINK			  READ WRITE NO
	 4 MIG26DTPUMP			  READ WRITE NO
	 6 MIG26REPLUG			  MOUNTED

alter pluggable database MIG26REPLUG open;

It will also take a couple of minutes for this upgrade to 26ai. You will see corresponding traces in the alert.log file of the 26ai container:

tail -f /u01/app/oracle/diag/rdbms/ctest26_s1/CTEST26/trace/alert_CTEST26.log
...
2026-01-12T11:34:21.075364+01:00
MIG26REPLUG(6):Starting Upgrade on PDB Open
...
MIG26REPLUG(6):***************************************************************
MIG26REPLUG(6):WARNING: Pluggable Database MIG26REPLUG with pdb id - 6 is
MIG26REPLUG(6):         altered with errors or warnings. Please look into
MIG26REPLUG(6):         PDB_PLUG_IN_VIOLATIONS view for more details.
MIG26REPLUG(6):***************************************************************
MIG26REPLUG(6):--ATTENTION--
MIG26REPLUG(6):Errors reported while opening PDB (ContainerId: 6) and have been recorded in pdb_alert$ table.
MIG26REPLUG(6) Error Violation: SQL Patch, Cause: '23.26.0.0.0 Release_Update 2509270520' is installed in the CDB but no release updates are installed in the PDB, Action: Call datapatch to install in the PDB or the CDB
2026-01-12T11:49:56.836062+01:00
MIG26REPLUG(6):Opening pdb with no Resource Manager plan active
Violations: Type: 1, Count: 1
Violations: Type: 2, Count: 1
Completed: Pluggable database MIG26REPLUG opened read write
Completed: alter pluggable database MIG26REPLUG open

Let’s have a look at my PDBs and apply the datapatch on MIG26REPLUG:

show pdbs
    CON_ID CON_NAME			  OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
	 2 PDB$SEED			  READ ONLY  NO
	 3 MIG26DBLINK			  READ WRITE NO
	 4 MIG26DTPUMP			  READ WRITE NO
	 6 MIG26REPLUG			  READ WRITE YES

exit

$ORACLE_HOME/OPatch/datapatch -pdbs MIG26REPLUG

SQL Patching tool version 23.26.0.0.0 Lim on Mon Jan 12 11:51:42 2026
Copyright (c) 2012, 2025, Oracle.  All rights reserved.

Log file for this invocation: /u01/app/oracle/product/23.0.0.0/dbhome_1/cfgtoollogs/sqlpatch/sqlpatch_sid_CTEST26_ts_2026_01_12_11_51_42_pid_24026/sqlpatch_invocation.log

Connecting to database...OK
Gathering database info...done

Note:  Datapatch will only apply or rollback SQL fixes for PDBs
       that are in an open state, no patches will be applied to closed PDBs.
       Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
       (Doc ID 1585822.1)

Bootstrapping registry and package to current versions...done
Determining current state...done

Current state of interim SQL patches:
  No interim patches found

Current state of release update SQL patches:
  Binary registry:
    23.26.0.0.0 Release_Update 250927052030: Installed
  PDB MIG26REPLUG:
    Applied 23.26.0.0.0 Release_Update 250927052030 successfully

Adding patches to installation queue and performing prereq checks...done
Installation queue:
  For the following PDBs: MIG26REPLUG
    No interim patches need to be rolled back
    No release update patches need to be installed
    No interim patches need to be applied

Bypass install queue:
  For the following PDBs: MIG26REPLUG
    No interim rollbacks will bypass install
    Patch 38404116 (Database Release Update : 23.26.0.0.0 (38404116) Gold Image): will bypass install
      Apply from 23.26.0.0.0 Release_Update 250927052030 to 23.26.0.0.0 Release_Update 250927052030
    No interim applys will bypass install


Installation queue after removing bypass entries...
Installation queue:
  For the following PDBs: MIG26REPLUG
    No interim patches need to be rolled back
    No release update patches need to be installed
    No interim patches need to be applied


Processing bypass install queue:
  Patch 38404116 apply (pdb MIG26REPLUG): SUCCESS (bypass_install)

SQL Patching tool complete on Mon Jan 12 11:51:50 2026

Now let’s restart the PDB in normal mode:

sqlplus / as sysdba

alter pluggable database MIG26REPLUG close immediate;
alter pluggable database MIG26REPLUG open;
alter pluggable database MIG26REPLUG save state;

show pdbs

    CON_ID CON_NAME			  OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
	 2 PDB$SEED			  READ ONLY  NO
	 3 MIG26DBLINK			  READ WRITE NO
	 4 MIG26DTPUMP			  READ WRITE NO
	 6 MIG26REPLUG			  READ WRITE NO

This PDB is now running on 26ai.

Let’s check my consultants table:

alter session set container=MIG26REPLUG;
select start_date from dbi.consultants where name like 'DUB%';
START_DAT
---------
04-SEP-17

exit

Pros & cons

+ Easy migration
+ Preserve source files with the copy option
+ Rather fast
– Need a downtime of the source PDB
– Reuse of source database is only possible by unplugging and plugging back the PDB with the xml file

Conclusion

There are multiple ways of migrating to 26ai. There is quite a lot of new features coming with 26ai. You will use them or not, but trying to migrate to this version is something to start thinking about in 2026.

L’article Migrate your database to 26ai on ODA est apparu en premier sur dbi Blog.

How to patch your ODA to 19.29?

Jérôme Dubar — Mon, 08 Dec 2025 16:32:04 +0000

Introduction

Patch 19.29 is now available for Oracle Database Appliance series. Let’s find out what’s new and how to apply this patch.

What’s new?

The most important new component is probably database 26ai (as a DB System only). But don’t get fooled, 26ai is only the new name of 23ai (23.26). It doesn’t matter, and this is nice to see this latest version coming to the ODA. 26ai is also coming to all on-premise systems.

The other change that comes with this version is the way of applying the system patch. Several years ago, you should have applied the server patch and the GI patch separately. It was then grouped within the same update-server command we’re using since years. Now the split makes a come back and update-server is replaced by update-servercomponents for system stuff, and update-gihome for GI. Also note that updade-dcsagent vanished in 19.27.

Improvements have also been made on the security side, with SE Linux now being enabled. Regarding DB Systems, CPU and memory allocation is now more flexible, this is great. Note that a DB System is no more limited to 1 container database since 19.23, making virtualization on ODA more appealing than before.

My overall feeling about this patch is “maturity and stability”. It’s all that we need for this kind of platform.

Which ODA is compatible with this 19.29 release?

The latest ODAs X11-HA, X11-L and X11-S are supported, as well as X10, X9-2 and X8-2 series. X7-2 series and older ones are not supported anymore. If you own one from these older generations, you should have a renewal plan for the coming months. I still recommend keeping your ODA 7 years, not less, not more. This blog post is still relevant today: https://www.dbi-services.com/blog/why-you-should-consider-keeping-your-oda-more-than-5-years/.

Is this patch a cumulative one?

The rule is now well established: you can apply a patch on top of the four previous ones. 19.29 can then be applied on top of 19.28, 19.27, 19.26 and 19.25. It’s why it makes sense patching once a year: this is the perfect balance between moderate security needs and ease of patching.

In my lab, I will use an ODA X8-2M running 19.28 with one DB home, one database and one DB System. This procedure should apply the same way on your ODA a soon as you’re using 19.25 or later.

Is there also a patch for my databases?

Only databases version 19c are supported for bare metal. You should be able to patch a 23ai database running as a DB System to 26ai, but you’d probably better deploying a new DB System and unplug/plug your PDBs into the brand new DB System.

Download the patch and clone files

These files are mandatory:

38427251 => the patch itself
30403673 => the GI clone needed for deploying newer 19c GI version
30403662 => the DB clone for deploying newer version of 19c

These files are optional:

30403643 => ISO file for reimaging, not needed for patching
36524660 => System image for 26ai DB Systems
36524627 => the GI clone needed for deploying new 26ai GI version
36524642 => the DB clone for deploying new 26ai DB version
32451228 => The newer system image for 19c DB Systems

Be sure to choose the very latest 19.29 when downloading some files, download link from MOS will first propose older versions for GI clones, DB clones and ISO files.

Prepare the patching

Before starting, please check these prerequisites:

filesystems /, /opt, /u01 and /root have at least 20% of available free space
additional manually installed rpms must be removed
revert profile scripts to default’s one (for grid and oracle users)
make sure you’ve planned a sufficient downtime (4+ hours depending on the number of databases and DB Systems)
do a sanity reboot before patching to kill zombie processes
use ODABR to make snapshots of the important filesystems prior patching: this tool is now included in the software distribution

Version precheck

Start to check current versions of the various components:

odacli describe-component
System Version
--------------
19.28.0.0.0

System Node Name
----------------
dbioda01

Local System Version
--------------------
19.28.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                      19.28.0.0.0          up-to-date

GI                                       19.28.0.0.250715     up-to-date

DB {
     OraDB19000_home7                    19.28.0.0.250715     up-to-date
     [CPROD19]
}

DCSCONTROLLER                            19.28.0.0.0          up-to-date

DCSCLI                                   19.28.0.0.0          up-to-date

DCSAGENT                                 19.28.0.0.0          up-to-date

DCSADMIN                                 19.28.0.0.0          up-to-date

OS                                       8.10                 up-to-date

ILOM                                     5.1.4.25.r160118     up-to-date

BIOS                                     52140100             up-to-date

LOCAL CONTROLLER FIRMWARE {
     [c4]                                8000D9AB             up-to-date
}

SHARED CONTROLLER FIRMWARE {
     [c0, c1]                            VDV1RL06             up-to-date
}

LOCAL DISK FIRMWARE {
     [c2d0, c2d1]                        XC311132             up-to-date
}

HMP                                      2.4.10.1.600         up-to-date

List the DB homes, databases, DB Systems and VMs:

odacli list-dbhomes
ID                                       Name                 DB Version           DB Edition Home Location                                            Status
---------------------------------------- -------------------- -------------------- ---------- -------------------------------------------------------- ----------
e120c4c9-91b9-47b4-a234-b8ada12fce69     OraDB19000_home7     19.28.0.0.250715     EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_7     CONFIGURED



odacli list-databases
ID                                       DB Name    DB Type  DB Version           CDB     Class    Edition  Shape    Storage  Status       DB Home ID               
---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
976a80f2-4653-469f-8cd4-ddc1a21aff51     CPROD19    SI       19.28.0.0.250715     true    OLTP     EE       odb8     ASM      CONFIGURED   e120c4c9-91b9-47b4-a234-b8ada12fce69


odacli list-dbsystems
Name                  Shape       GI version          DB info                         Status                  Created                   Updated
--------------------  ----------  ------------------  ------------------------------  ----------------------  ------------------------  ------------------------
dbs-03-tst            dbs2        19.28.0.0.250715    19.28(CONFIGURED=1)             CONFIGURED              2025-12-03 15:05:31 CET   2025-12-03 15:47:19 CET

odacli list-vms
No data found for resource VM.

Update the DCS components

Updating DCS components is the first step, after registering the patch file:

cd /opt/dbi
unzip p38427251_1929000_Linux-x86-64.zip

odacli update-repository -f /opt/dbi/oda-sm-19.29.0.0.0-251117-server.zip
sleep 30 ; odacli describe-job -i "91189839-e855-48ea-a6b1-7d9695da52a5"
Job details
----------------------------------------------------------------
                     ID:  7e69a05f-61fe-4b13-af5d-d78cfb7f11a9
            Description:  Repository Update
                 Status:  Success
                Created:  December 03, 2025 16:09:45 CET
                Message:  /opt/dbi/oda-sm-19.29.0.0.0-251117-server.zip

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Unzip bundle                             December 03, 2025 16:09:49 CET           December 03, 2025 16:10:00 CET           Success


odacli describe-component
System Version
--------------
19.28.0.0.0

System Node Name
----------------
dbioda01

Local System Version
--------------------
19.28.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                      19.28.0.0.0          19.29.0.0.0

GI                                       19.28.0.0.250715     19.29.0.0.251021

DB {
     OraDB19000_home7                    19.28.0.0.250715     19.29.0.0.251021
     [CPROD19]
}

DCSCONTROLLER                            19.28.0.0.0          19.29.0.0.0

DCSCLI                                   19.28.0.0.0          19.29.0.0.0

DCSAGENT                                 19.28.0.0.0          19.29.0.0.0

DCSADMIN                                 19.28.0.0.0          19.29.0.0.0

OS                                       8.10                 up-to-date

ILOM                                     5.1.4.25.r160118     5.1.5.22.r165351

BIOS                                     52140100             52160100

LOCAL CONTROLLER FIRMWARE {
     [c4]                                8000D9AB             up-to-date
}

SHARED CONTROLLER FIRMWARE {
     [c0, c1]                            VDV1RL06             up-to-date
}

LOCAL DISK FIRMWARE {
     [c2d0, c2d1]                        XC311132             up-to-date
}

HMP                                      2.4.10.1.600         up-to-date

Let’s update the DCS components to 19.29:

odacli update-dcsadmin -v 19.29.0.0.0

sleep 60 ; odacli describe-job -i "f2d216d5-f60d-46d6-a967-900c6e137421"
Job details
----------------------------------------------------------------
                     ID:  f2d216d5-f60d-46d6-a967-900c6e137421
            Description:  DcsAdmin patching to 19.29.0.0.0
                 Status:  Success
                Created:  December 03, 2025 16:12:26 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Dcs-admin upgrade                        December 03, 2025 16:12:26 CET           December 03, 2025 16:12:36 CET           Success
Ping DCS Admin                           December 03, 2025 16:12:36 CET           December 03, 2025 16:13:43 CET           Success



sleep 30 ;  odacli update-dcscomponents -v 19.29.0.0.0
{
  "jobId" : "5cd855d4-7b35-44dc-84f9-4625f84d461b",
  "status" : "Success",
  "message" : "Update-dcscomponents is successful on all the node(s): DCS-Agent shutdown is successful. MySQL upgrade is successful. Metadata schema update is done. Script '/opt/oracle/dcs/log/jobfiles/5cd855d4-7b35-44dc-84f9-4625f84d461b/apply_metadata_change.sh' ran successfully. dcsagent RPM upgrade is successful. dcscli RPM upgrade is successful. dcscontroller RPM upgrade is successful. ahf RPM upgrade is successful.  Successfully reset the Keystore password. HAMI RPM and DCS ensemble update was successful.  Skipped removing old Libs. Successfully ran setupAgentAuth.sh ",
  "reports" : null,
  "createTimestamp" : "December 03, 2025 16:14:32 PM CET",
  "description" : "Update-dcscomponents job completed and is not part of Agent job list",
  "updatedTime" : "December 03, 2025 16:19:11 PM CET",
  "jobType" : null,
  "externalRequestId" : null,
  "action" : null
}

System patching

Let’s do the prepatching of the system with the new -sc option:

odacli create-prepatchreport -sc -v 19.29.0.0.0

sleep 180 ; odacli describe-prepatchreport -i 317b0f75-fed7-480b-9dba-af7c635fabea

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  317b0f75-fed7-480b-9dba-af7c635fabea
            Description:  Patch pre-checks for [OS, ILOM, ORACHKSERVER, SERVER] to 19.29.0.0.0
                 Status:  SUCCESS
                Created:  December 3, 2025 4:20:13 PM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
dbioda01

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.29.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   No dependencies found for RPMs being
                                          removed, updated and installed. Check
                                          /opt/oracle/dcs/log/jobfiles/
                                          yumdryrunout_2025-12-03_16-20-
                                          29.0193.1_251.log file for more
                                          details
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.29.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate all KVM ACFS           Success   All KVM ACFS resources are running
resources are running
Validate DB System VM states    Success   All DB System VMs states are expected
Validate DB System AFD state    Success   All DB Systems are on required
                                          versions
Validate command execution      Success   Validated command execution

OK let’s do the system patch:

odacli update-servercomponents -v 19.29.0.0.0
...

The server will reboot at the end of the patching. Let’s then check the job:

odacli describe-job -i "7f52ba58-f0d5-4055-864b-caae4209ce29"

Job details
----------------------------------------------------------------
                     ID:  7f52ba58-f0d5-4055-864b-caae4209ce29
            Description:  Server Patching to 19.29.0.0.0
                 Status:  Success
                Created:  December 03, 2025 16:26:08 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Deactivate Unit[dnf-makecache.timer]     December 03, 2025 16:26:14 CET           December 03, 2025 16:26:15 CET           Success
Validate ILOM server reachable           December 03, 2025 16:26:14 CET           December 03, 2025 16:26:14 CET           Success
Validating GI user metadata              December 03, 2025 16:26:14 CET           December 03, 2025 16:26:14 CET           Success
Deactivate Unit[kdump.service]           December 03, 2025 16:26:15 CET           December 03, 2025 16:26:15 CET           Success
Modify BM udev rules                     December 03, 2025 16:26:15 CET           December 03, 2025 16:26:35 CET           Success
Stop oakd                                December 03, 2025 16:26:35 CET           December 03, 2025 16:26:39 CET           Success
Creating repositories using yum          December 03, 2025 16:26:39 CET           December 03, 2025 16:26:41 CET           Success
Updating YumPluginVersionLock rpm        December 03, 2025 16:26:41 CET           December 03, 2025 16:26:44 CET           Success
Applying OS Patches                      December 03, 2025 16:26:44 CET           December 03, 2025 16:32:20 CET           Success
Applying HMP Patches                     December 03, 2025 16:32:20 CET           December 03, 2025 16:32:23 CET           Success
Creating repositories using yum          December 03, 2025 16:32:20 CET           December 03, 2025 16:32:20 CET           Success
Oda-hw-mgmt upgrade                      December 03, 2025 16:32:23 CET           December 03, 2025 16:32:52 CET           Success
Patch location validation                December 03, 2025 16:32:23 CET           December 03, 2025 16:32:23 CET           Success
Setting SELinux mode                     December 03, 2025 16:32:23 CET           December 03, 2025 16:32:23 CET           Success
Applying Firmware local Disk Patches     December 03, 2025 16:32:53 CET           December 03, 2025 16:32:57 CET           Success
OSS Patching                             December 03, 2025 16:32:53 CET           December 03, 2025 16:32:53 CET           Success
Applying Firmware local Controller Patch December 03, 2025 16:32:57 CET           December 03, 2025 16:33:01 CET           Success
Checking Ilom patch Version              December 03, 2025 16:33:01 CET           December 03, 2025 16:33:01 CET           Success
Patch location validation                December 03, 2025 16:33:01 CET           December 03, 2025 16:33:01 CET           Success
Save password in Wallet                  December 03, 2025 16:33:01 CET           December 03, 2025 16:33:02 CET           Success
Apply Ilom patch                         December 03, 2025 16:33:02 CET           December 03, 2025 16:43:49 CET           Success
Disabling IPMI v2                        December 03, 2025 16:33:02 CET           December 03, 2025 16:33:02 CET           Success
Copying Flash Bios to Temp location      December 03, 2025 16:43:49 CET           December 03, 2025 16:43:49 CET           Success
Start oakd                               December 03, 2025 16:43:49 CET           December 03, 2025 16:44:06 CET           Success
Add SYSNAME in Env                       December 03, 2025 16:44:06 CET           December 03, 2025 16:44:06 CET           Success
Cleanup JRE Home                         December 03, 2025 16:44:06 CET           December 03, 2025 16:44:06 CET           Success
Starting the clusterware                 December 03, 2025 16:44:06 CET           December 03, 2025 16:45:42 CET           Success
Generating and saving BOM                December 03, 2025 16:45:42 CET           December 03, 2025 16:46:12 CET           Success
Update System version                    December 03, 2025 16:45:42 CET           December 03, 2025 16:45:42 CET           Success
Update lvm.conf file                     December 03, 2025 16:45:42 CET           December 03, 2025 16:45:42 CET           Success
PreRebootNode Actions                    December 03, 2025 16:46:12 CET           December 03, 2025 16:47:17 CET           Success
Reboot Node                              December 03, 2025 16:47:17 CET           December 03, 2025 16:47:17 CET           Success

GI patching

Let’s unzip and register the patch file, and do the precheck for GI:

cd /opt/dbi
unzip -o p30403673_1929000_Linux-x86-64.zip

odacli update-repository -f /opt/dbi/odacli-dcs-19.29.0.0.0-251117-GI-19.29.0.0.zip
sleep 30 ; odacli describe-job -i "2e15156c-451b-470f-a523-03c4d024b726"

Job details
----------------------------------------------------------------
                     ID:  2e15156c-451b-470f-a523-03c4d024b726
            Description:  Repository Update
                 Status:  Success
                Created:  December 03, 2025 17:29:43 CET
                Message:  /opt/dbi/odacli-dcs-19.29.0.0.0-251117-GI-19.29.0.0.zip

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Unzip bundle                             December 03, 2025 17:29:45 CET           December 03, 2025 17:30:32 CET           Success


odacli create-prepatchreport -gi -v 19.29.0.0.0

sleep 180 ; odacli describe-prepatchreport -i be3c7eb0-7bd8-4295-aa7f-ecd9e104d66f

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  be3c7eb0-7bd8-4295-aa7f-ecd9e104d66f
            Description:  Patch pre-checks for [RHPGI, GI] to 19.29.0.0.0
                 Status:  SUCCESS
                Created:  December 3, 2025 5:30:45 PM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
dbioda01

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__RHPGI__
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Successfully validated minimum version
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.29.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Validate space for clones       Success   Clones volume is already created
volume
Validate command execution      Success   Validated command execution

Let’s apply the GI update now:

odacli update-gihome -v 19.29.0.0.0
sleep 400 ;  odacli describe-job -i "5eaaa4ad-996b-4024-858d-a0f0082705d5"

Job details
----------------------------------------------------------------
                     ID:  5eaaa4ad-996b-4024-858d-a0f0082705d5
            Description:  Patch GI with RHP to 19.29.0.0.0
                 Status:  Success
                Created:  December 03, 2025 17:37:30 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Starting the clusterware                 December 03, 2025 17:37:44 CET           December 03, 2025 17:37:44 CET           Success
Creating GI home directories             December 03, 2025 17:37:45 CET           December 03, 2025 17:37:45 CET           Success
Extract GI clone                         December 03, 2025 17:37:45 CET           December 03, 2025 17:37:45 CET           Success
Provisioning Software Only GI with RHP   December 03, 2025 17:37:45 CET           December 03, 2025 17:37:45 CET           Success
Registering image                        December 03, 2025 17:37:45 CET           December 03, 2025 17:37:45 CET           Success
Registering image                        December 03, 2025 17:37:45 CET           December 03, 2025 17:37:45 CET           Success
Registering working copy                 December 03, 2025 17:37:45 CET           December 03, 2025 17:37:45 CET           Success
Patch GI with RHP                        December 03, 2025 17:38:21 CET           December 03, 2025 17:43:07 CET           Success
Set CRS ping target                      December 03, 2025 17:43:07 CET           December 03, 2025 17:43:07 CET           Success
Updating .bashrc                         December 03, 2025 17:43:07 CET           December 03, 2025 17:43:07 CET           Success
Updating GI home metadata                December 03, 2025 17:43:07 CET           December 03, 2025 17:43:07 CET           Success
Updating GI home version                 December 03, 2025 17:43:07 CET           December 03, 2025 17:43:12 CET           Success
Updating All DBHome version              December 03, 2025 17:43:12 CET           December 03, 2025 17:43:17 CET           Success
Starting the clusterware                 December 03, 2025 17:43:38 CET           December 03, 2025 17:43:39 CET           Success
Validate ACFS resources are running      December 03, 2025 17:43:39 CET           December 03, 2025 17:43:39 CET           Success
Validate DB System VMs states            December 03, 2025 17:43:39 CET           December 03, 2025 17:43:40 CET           Success
Validate GI availability                 December 03, 2025 17:43:39 CET           December 03, 2025 17:43:39 CET           Success
Patch CPU Pools distribution             December 03, 2025 17:43:40 CET           December 03, 2025 17:43:40 CET           Success
Patch DB System domain config            December 03, 2025 17:43:40 CET           December 03, 2025 17:43:40 CET           Success
Patch KVM CRS type                       December 03, 2025 17:43:40 CET           December 03, 2025 17:43:40 CET           Success
Patch VM vDisks CRS dependencies         December 03, 2025 17:43:40 CET           December 03, 2025 17:43:40 CET           Success
Save custom VNetworks to storage         December 03, 2025 17:43:40 CET           December 03, 2025 17:43:41 CET           Success
Add network filters to DB Systems        December 03, 2025 17:43:41 CET           December 03, 2025 17:43:41 CET           Success
Create network filters                   December 03, 2025 17:43:41 CET           December 03, 2025 17:43:41 CET           Success
Patch DB Systems vDisks CRS dependencies December 03, 2025 17:43:41 CET           December 03, 2025 17:43:42 CET           Success
Patch DB Systems custom scale metadata   December 03, 2025 17:43:42 CET           December 03, 2025 17:43:42 CET           Success

No reboot is needed for this patch.

Check the versions

odacli describe-component
System Version
--------------
19.29.0.0.0

System Node Name
----------------
dbioda01

Local System Version
--------------------
19.29.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                      19.29.0.0.0          up-to-date

GI                                       19.29.0.0.251021     up-to-date

DB {
     OraDB19000_home7                    19.28.0.0.250715     19.29.0.0.251021
     [CPROD19]
}

DCSCONTROLLER                            19.29.0.0.0          up-to-date

DCSCLI                                   19.29.0.0.0          up-to-date

DCSAGENT                                 19.29.0.0.0          up-to-date

DCSADMIN                                 19.29.0.0.0          up-to-date

OS                                       8.10                 up-to-date

ILOM                                     5.1.5.22.r165351     up-to-date

BIOS                                     52160100             up-to-date

LOCAL CONTROLLER FIRMWARE {
     [c4]                                8000D9AB             up-to-date
}

SHARED CONTROLLER FIRMWARE {
     [c0, c1]                            VDV1RL06             up-to-date
}

LOCAL DISK FIRMWARE {
     [c2d0, c2d1]                        XC311132             up-to-date
}

HMP                                      2.4.10.1.600         up-to-date

Patching the storage

Patching the storage is only needed if describe-component tells you that you’re not up-to-date. On my X8-2M, it wasn’t needed. If your ODA needs the storage patch, it’s easy:

odacli update-storage -v 19.29.0.0.0
odacli describe-job -i ...

The server will reboot when done.

Patching the DB homes

It’s now time to patch the DB home and the database on my ODA. Let’s first unzip and register the patch file in the repository:

unzip -o p30403662_1929000_Linux-x86-64.zip
odacli update-repository -f /opt/dbi/odacli-dcs-19.29.0.0.0-251117-DB-19.29.0.0.zip 
sleep 30; odacli describe-job -i "480c3911-d673-47fd-b6c5-f65b2cc4a1b9"

Job details
----------------------------------------------------------------
                     ID:  480c3911-d673-47fd-b6c5-f65b2cc4a1b9
            Description:  Repository Update
                 Status:  Success
                Created:  December 03, 2025 17:51:13 CET
                Message:  /opt/dbi/odacli-dcs-19.29.0.0.0-251117-DB-19.29.0.0.zip

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Unzip bundle                             December 03, 2025 17:51:13 CET           December 03, 2025 17:51:49 CET           Success



odacli list-dbhomes
ID                                       Name                 DB Version           DB Edition Home Location                                            Status
---------------------------------------- -------------------- -------------------- ---------- -------------------------------------------------------- ----------
e120c4c9-91b9-47b4-a234-b8ada12fce69     OraDB19000_home7     19.28.0.0.250715     EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_7     CONFIGURED

Let’s check if the patch can be applied, then patch this DB home:

odacli create-prepatchreport -d -i e120c4c9-91b9-47b4-a234-b8ada12fce69 -v 19.29.0.0.0

sleep 600; odacli describe-prepatchreport -i fed289b2-848a-460f-9ba7-ef87c2a08dca

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  fed289b2-848a-460f-9ba7-ef87c2a08dca
            Description:  Patch pre-checks for [DB, RHPDB, ORACHKDB] to 19.29.0.0.0: DbHome is OraDB19000_home7
                 Status:  SUCCESS
                Created:  December 3, 2025 5:53:51 PM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
dbioda01

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__DB__
Validate DB Home ID             Success   Validated DB Home ID:
                                          e120c4c9-91b9-47b4-a234-b8ada12fce69
Validate patching tag           Success   Validated patching tag: 19.29.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate minimum agent version  Success   Validated minimum agent version
Is GI upgraded                  Success   Validated GI is upgraded
Validate available space for    Success   Validated free space required under
db                                        /u01
Validate there is usable        Success   Successfully validated Oracle Base
space under oracle base                   usable space
Validate glogin.sql file        Success   Successfully verified glogin.sql
                                          won't break patching
Validate dbHomesOnACFS          Success   User has configured disk group for
configured                                Database homes on ACFS
Validate Oracle base            Success   Successfully validated Oracle Base
Is DB clone available           Success   Successfully validated clone file
                                          exists
Validate command execution      Success   Validated command execution

__RHPDB__
Evaluate DBHome patching with   Success   Successfully validated updating
RHP                                       dbhome with RHP.  and local patching
                                          is possible
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

odacli update-dbhome -i e120c4c9-91b9-47b4-a234-b8ada12fce69 -v 19.29.0.0.0

sleep 600;  odacli describe-job -i "b2676a55-96de-4101-8686-98c6a88b8477"
Job details
----------------------------------------------------------------
                     ID:  b2676a55-96de-4101-8686-98c6a88b8477
            Description:  DB Home Patching to 19.29.0.0.0: Home ID is e120c4c9-91b9-47b4-a234-b8ada12fce69
                 Status:  Success
                Created:  December 03, 2025 18:05:45 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Creating wallet for DB Client            December 03, 2025 18:06:34 CET           December 03, 2025 18:06:34 CET           Success
Patch databases by RHP - [CPROD19]       December 03, 2025 18:06:34 CET           December 03, 2025 18:13:06 CET           Success
Updating database metadata               December 03, 2025 18:13:06 CET           December 03, 2025 18:13:06 CET           Success
Upgrade pwfile to 12.2                   December 03, 2025 18:13:06 CET           December 03, 2025 18:13:09 CET           Success
Set log_archive_dest for Database        December 03, 2025 18:13:09 CET           December 03, 2025 18:13:12 CET           Success
Populate PDB metadata                    December 03, 2025 18:13:13 CET           December 03, 2025 18:13:14 CET           Success
Generating and saving BOM                December 03, 2025 18:13:15 CET           December 03, 2025 18:13:54 CET           Success
TDE parameter update                     December 03, 2025 18:14:24 CET           December 03, 2025 18:14:24 CET           Success

Let’s check if everything is fine:

odacli list-dbhomes
ID                                       Name                 DB Version           DB Edition Home Location                                            Status
---------------------------------------- -------------------- -------------------- ---------- -------------------------------------------------------- ----------
e120c4c9-91b9-47b4-a234-b8ada12fce69     OraDB19000_home7     19.28.0.0.250715     EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_7     CONFIGURED
57c0dd7f-dcf4-4a38-9e79-4bf8c78e81bb     OraDB19000_home9     19.29.0.0.251021     EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_9     CONFIGURED

odacli list-databases
ID                                       DB Name    DB Type  DB Version           CDB     Class    Edition  Shape    Storage  Status       DB Home ID               
---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
976a80f2-4653-469f-8cd4-ddc1a21aff51     CPROD19    SI       19.29.0.0.251021     true    OLTP     EE       odb8     ASM      CONFIGURED   57c0dd7f-dcf4-4a38-9e79-4bf8c78e81bb

Let’s now remove the old DB home:

odacli delete-dbhome -i e120c4c9-91b9-47b4-a234-b8ada12fce69
...

Cleanse the old patches

Don’t forget to remove the previous patch from the repository:

odacli cleanup-patchrepo -comp all -v 19.28.0.0.0

odacli describe-job -i "76ba3e95-bb71-4ebe-b7b2-f3cac07d89dd"
Job details
----------------------------------------------------------------
                     ID:  76ba3e95-bb71-4ebe-b7b2-f3cac07d89dd
            Description:  Cleanup patchrepos
                 Status:  Success
                Created:  December 03, 2025 18:19:22 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Cleanup Repository                       December 03, 2025 18:19:22 CET           December 03, 2025 18:19:22 CET           Success
Cleanup old ASR rpm                      December 03, 2025 18:19:22 CET           December 03, 2025 18:19:22 CET           Success

Old GI binaries are still there, it’s better removing them manually:

du -hs /u01/app/19.2*
14G     /u01/app/19.28.0.0
14G     /u01/app/19.29.0.0

rm -rf /u01/app/19.28.0.0

Post-patching tasks

You will need to put back your specific configuration:

add your additional RPMs
put back your profile scripts for grid and oracle users
check if monitoring still works

Patching the DB System

If you use DB Systems on your ODA, meaning that some of your databases are running in dedicated VMs, you will need to apply the patch inside each DB System. As the repository is shared, patch files are already available for DB Systems. Applying the patch is similar to what you did on bare metal:

ssh dbs-03-tst

odacli update-dcsadmin -v 19.29.0.0.0

sleep 60 ; odacli describe-job -i 43df9afb-adc1-479c-8987-c8d24f056c02

Job details
----------------------------------------------------------------
                     ID:  43df9afb-adc1-479c-8987-c8d24f056c02
            Description:  DcsAdmin patching to 19.29.0.0.0
                 Status:  Success
                Created:  December 08, 2025 10:15:13 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Dcs-admin upgrade                        December 08, 2025 10:15:14 CET           December 08, 2025 10:15:25 CET           Success
Ping DCS Admin                           December 08, 2025 10:15:25 CET           December 08, 2025 10:16:33 CET           Success

sleep 30 ;  odacli update-dcscomponents -v 19.29.0.0.0
{
  "jobId" : "91fd5f67-5ba8-4636-9046-0fe1921a659e",
  "status" : "Success",
  "message" : "Update-dcscomponents is successful on all the node(s): DCS-Agent shutdown is successful. MySQL upgrade is successful. Metadata schema update is done. Script '/opt/oracle/dcs/log/jobfiles/91fd5f67-5ba8-4636-9046-0fe1921a659e/apply_metadata_change.sh' ran successfully. dcsagent RPM upgrade is successful. dcscli RPM upgrade is successful. dcscontroller RPM upgrade is successful. ahf RPM upgrade is successful.  Successfully reset the Keystore password. HAMI RPM and DCS ensemble update was successful.  Skipped removing old Libs. Successfully ran setupAgentAuth.sh ",
  "reports" : null,
  "createTimestamp" : "December 08, 2025 10:17:38 AM CET",
  "description" : "Update-dcscomponents job completed and is not part of Agent job list",
  "updatedTime" : "December 08, 2025 10:23:37 AM CET",
  "jobType" : null,
  "externalRequestId" : null,
  "action" : null
}

odacli create-prepatchreport -sc -v 19.29.0.0.0

sleep 20 ; odacli describe-prepatchreport -i f2f90339-16b8-49a5-be8c-408dd0e9f28b
ps -ef | grep pmon

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  f2f90339-16b8-49a5-be8c-408dd0e9f28b
            Description:  Patch pre-checks for [OS, ORACHKSERVER, SERVER] to 19.29.0.0.0
                 Status:  SUCCESS
                Created:  December 8, 2025 10:25:17 AM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
dbs-03-tst

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.29.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   No dependencies found for RPMs being
                                          removed, updated and installed. Check
                                          /opt/oracle/dcs/log/jobfiles/
                                          yumdryrunout_2025-12-08_10-25-
                                          34.0670.1_222.log file for more
                                          details
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate all KVM ACFS           Success   All KVM ACFS resources are running
resources are running
Validate DB System VM states    Success   All DB System VMs states are expected
Enable support for Multi-DB     Success   No need to convert the DB System
Validate DB System AFD state    Success   AFD is not configured
Validate command execution      Success   Validated command execution

odacli update-servercomponents -v 19.29.0.0.0

The DB System will reboot.

odacli describe-job -i 5a23ae5b-43ed-4c39-ba79-21cd8a125b79

Job details
----------------------------------------------------------------
                     ID:  5a23ae5b-43ed-4c39-ba79-21cd8a125b79
            Description:  Server Patching to 19.29.0.0.0
                 Status:  Success
                Created:  December 08, 2025 10:30:04 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Validating GI user metadata              December 08, 2025 10:30:13 CET           December 08, 2025 10:30:13 CET           Success
Deactivate Unit[dnf-makecache.timer]     December 08, 2025 10:30:14 CET           December 08, 2025 10:30:14 CET           Success
Deactivate Unit[kdump.service]           December 08, 2025 10:30:14 CET           December 08, 2025 10:30:15 CET           Success
Modify DBVM udev rules                   December 08, 2025 10:30:15 CET           December 08, 2025 10:30:36 CET           Success
Creating repositories using yum          December 08, 2025 10:30:36 CET           December 08, 2025 10:30:39 CET           Success
Updating YumPluginVersionLock rpm        December 08, 2025 10:30:39 CET           December 08, 2025 10:30:42 CET           Success
Applying OS Patches                      December 08, 2025 10:30:42 CET           December 08, 2025 10:34:11 CET           Success
Creating repositories using yum          December 08, 2025 10:34:11 CET           December 08, 2025 10:34:12 CET           Success
Applying HMP Patches                     December 08, 2025 10:34:12 CET           December 08, 2025 10:34:15 CET           Success
Setting SELinux mode                     December 08, 2025 10:34:15 CET           December 08, 2025 10:34:15 CET           Success
Oda-hw-mgmt upgrade                      December 08, 2025 10:34:16 CET           December 08, 2025 10:34:44 CET           Success
Patch location validation                December 08, 2025 10:34:16 CET           December 08, 2025 10:34:16 CET           Success
Cleanup JRE Home                         December 08, 2025 10:34:45 CET           December 08, 2025 10:34:45 CET           Success
Generating and saving BOM                December 08, 2025 10:34:56 CET           December 08, 2025 10:35:09 CET           Success
Update System version                    December 08, 2025 10:34:56 CET           December 08, 2025 10:34:56 CET           Success
PreRebootNode Actions                    December 08, 2025 10:35:09 CET           December 08, 2025 10:35:09 CET           Success
Reboot Node                              December 08, 2025 10:35:09 CET           December 08, 2025 10:35:09 CET           Success

odacli create-prepatchreport -gi -v 19.29.0.0.0

sleep 240 ; odacli describe-prepatchreport -i 56c7b4b1-3787-42af-b4b0-0fa6715a91f7

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  56c7b4b1-3787-42af-b4b0-0fa6715a91f7
            Description:  Patch pre-checks for [RHPGI, GI] to 19.29.0.0.0
                 Status:  SUCCESS
                Created:  December 8, 2025 10:37:05 AM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
dbs-03-tst

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__RHPGI__
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Successfully validated minimum version
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.29.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate BM versions            Success   Validated BM server components
                                          versions
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate command execution      Success   Validated command execution

odacli update-gihome -v 19.29.0.0.0

sleep 600 ; odacli describe-job -i 571205f2-bdf2-43a2-944a-ec2765148446

odacli describe-job -i 571205f2-bdf2-43a2-944a-ec2765148446

Job details
----------------------------------------------------------------
                     ID:  571205f2-bdf2-43a2-944a-ec2765148446
            Description:  Patch GI with RHP to 19.29.0.0.0
                 Status:  Success
                Created:  December 08, 2025 10:43:47 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Starting the clusterware                 December 08, 2025 10:44:07 CET           December 08, 2025 10:44:07 CET           Success
Registering image                        December 08, 2025 10:44:08 CET           December 08, 2025 10:44:08 CET           Success
Registering working copy                 December 08, 2025 10:44:08 CET           December 08, 2025 10:44:09 CET           Success
Creating GI home directories             December 08, 2025 10:44:09 CET           December 08, 2025 10:44:09 CET           Success
Extract GI clone                         December 08, 2025 10:44:09 CET           December 08, 2025 10:44:09 CET           Success
Provisioning Software Only GI with RHP   December 08, 2025 10:44:09 CET           December 08, 2025 10:44:09 CET           Success
Registering image                        December 08, 2025 10:44:09 CET           December 08, 2025 10:44:09 CET           Success
Patch GI with RHP                        December 08, 2025 10:44:49 CET           December 08, 2025 10:49:16 CET           Success
Set CRS ping target                      December 08, 2025 10:49:16 CET           December 08, 2025 10:49:16 CET           Success
Updating .bashrc                         December 08, 2025 10:49:16 CET           December 08, 2025 10:49:16 CET           Success
Updating GI home metadata                December 08, 2025 10:49:16 CET           December 08, 2025 10:49:17 CET           Success
Updating GI home version                 December 08, 2025 10:49:17 CET           December 08, 2025 10:49:24 CET           Success
Updating All DBHome version              December 08, 2025 10:49:24 CET           December 08, 2025 10:49:30 CET           Success
Patch DB System on BM                    December 08, 2025 10:50:05 CET           December 08, 2025 10:50:11 CET           Success
Starting the clusterware                 December 08, 2025 10:50:05 CET           December 08, 2025 10:50:05 CET           Success


odacli list-dbhomes
ID                                       Name                 DB Version           DB Edition Home Location                                            Status
---------------------------------------- -------------------- -------------------- ---------- -------------------------------------------------------- ----------
46268d88-e958-4c16-b45b-c32d5e0203fb     OraDB19000_home1     19.28.0.0.250715     EE         /u01/app/oracle/product/19.0.0.0/dbhome_1                CONFIGURED

odacli create-prepatchreport -d -i 46268d88-e958-4c16-b45b-c32d5e0203fb -v 19.29.0.0.0

sleep 600 ;  odacli describe-prepatchreport -i eb00906a-0ecc-4a9d-968a-272d4c3719f4

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  eb00906a-0ecc-4a9d-968a-272d4c3719f4
            Description:  Patch pre-checks for [DB, RHPDB, ORACHKDB] to 19.29.0.0.0: DbHome is OraDB19000_home1
                 Status:  SUCCESS
                Created:  December 8, 2025 11:27:25 AM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
dbs-03-tst

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__DB__
Validate DB Home ID             Success   Validated DB Home ID:
                                          46268d88-e958-4c16-b45b-c32d5e0203fb
Validate patching tag           Success   Validated patching tag: 19.29.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate minimum agent version  Success   Validated minimum agent version
Is GI upgraded                  Success   Validated GI is upgraded
Validate available space for    Success   Validated free space required under
db                                        /u01
Validate there is usable        Success   Successfully validated Oracle Base
space under oracle base                   usable space
Validate glogin.sql file        Success   Successfully verified glogin.sql
                                          won't break patching
Is DB clone available           Success   Successfully validated clone file
                                          exists
Validate command execution      Success   Validated command execution

__RHPDB__
Evaluate DBHome patching with   Success   Successfully validated updating
RHP                                       dbhome with RHP.  and local patching
                                          is possible
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution


odacli update-dbhome -i 46268d88-e958-4c16-b45b-c32d5e0203fb -v 19.29.0.0.0

sleep 600 ; odacli describe-job -i aac90798-a6a4-4740-bfa6-77bcb80cba7c

Job details
----------------------------------------------------------------
                     ID:  aac90798-a6a4-4740-bfa6-77bcb80cba7c
            Description:  DB Home Patching to 19.29.0.0.0: Home ID is 46268d88-e958-4c16-b45b-c32d5e0203fb
                 Status:  Success
                Created:  December 08, 2025 11:36:57 CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Creating wallet for DB Client            December 08, 2025 11:37:43 CET           December 08, 2025 11:37:43 CET           Success
Patch databases by RHP - [CTEST19]       December 08, 2025 11:37:43 CET           December 08, 2025 11:45:11 CET           Success
Updating database metadata               December 08, 2025 11:45:11 CET           December 08, 2025 11:45:12 CET           Success
Upgrade pwfile to 12.2                   December 08, 2025 11:45:12 CET           December 08, 2025 11:45:17 CET           Success
Set log_archive_dest for Database        December 08, 2025 11:45:17 CET           December 08, 2025 11:45:21 CET           Success
Populate PDB metadata                    December 08, 2025 11:45:22 CET           December 08, 2025 11:45:24 CET           Success
Generating and saving BOM                December 08, 2025 11:45:24 CET           December 08, 2025 11:46:10 CET           Success
TDE parameter update                     December 08, 2025 11:46:39 CET           December 08, 2025 11:46:39 CET           Success

odacli list-databases
ID                                       DB Name    DB Type  DB Version           CDB     Class    Edition  Shape    Storage  Status       DB Home ID
---------------------------------------- ---------- -------- -------------------- ------- -------- -------- -------- -------- ------------ ----------------------------------------
54e88627-a3cf-4696-956b-6262bbd51cf0     CTEST19    SI       19.29.0.0.251021     true    OLTP     EE       odb2     ASM      CONFIGURED   85b6e4eb-5db4-4165-bfb3-e3da52dc4777

odacli delete-dbhome -i 46268d88-e958-4c16-b45b-c32d5e0203fb
...

odacli describe-component
System Version
--------------
19.29.0.0.0

System Node Name
----------------
dbs-03-tst

Local System Version
--------------------
19.29.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                      19.29.0.0.0          up-to-date

GI                                       19.29.0.0.251021     up-to-date

DB {
     OraDB19000_home2                    19.29.0.0.251021     up-to-date
     [CTEST19]
}

DCSCONTROLLER                            19.29.0.0.0          up-to-date

DCSCLI                                   19.29.0.0.0          up-to-date

DCSAGENT                                 19.29.0.0.0          up-to-date

DCSADMIN                                 19.29.0.0.0          up-to-date

OS                                       8.10                 up-to-date

Don’t forget to apply this procedure on each of your DB Systems.

Provision a new 26ai DB System

This is an optional step, only if you’d like to try 26ai database. First unzip and register the VM template, GI 26ai and DB 26ai:

unzip -o p36524660_1929000_Linux-x86-64.zip
unzip -o p36524627_1929000_Linux-x86-64.zip
unzip -o p36524642_1929000_Linux-x86-64.zip

odacli update-repository -f /opt/dbi/odacli-dcs-23.26.0.0.0-251116-ODAVM-19.29.0.0.zip
odacli update-repository -f /opt/dbi/odacli-dcs-23.26.0.0.0-251116-GI-23.26.0.0.zip
odacli update-repository -f /opt/dbi/odacli-dcs-23.26.0.0.0-251116-DB-23.26.0.0.zip

sleep 30 ; odacli list-jobs | tail -n 4
d4700315-db8c-4522-af55-0fddd262bfe4     Repository Update                                                           2025-12-08 15:20:14 CET             Success
ba7d452f-e03d-46d0-a607-fd7c758cd1b1     Repository Update                                                           2025-12-08 15:20:59 CET             Success
bfc102d1-985b-4792-8054-03709aa8d949     Repository Update                                                           2025-12-08 15:21:20 CET             Success

odacli describe-dbsystem-image | grep 23.26
DBVM                  23.26.0.0.0           23.26.0.0.0
GI                    23.26.0.0.0           23.26.0.0.0
DB                    23.26.0.0.0           23.26.0.0.0

Now let’s create a json file based on the one I used to create my 19.28 DB System and adjust some parameters. Then create the DB System:

cat create_dbs-03-tst-cdb.json | sed 's/dbs-03-tst/dbs-04-tst/g' | sed 's/10.16.0.146/10.16.0.147/g' | sed 's/CTEST19/CTEST26/g' | sed 's/19.28.0.0.250715/23.26.0.0.0/g' > create_dbs-04-tst-cdb.json

odacli create-dbsystem -p /opt/dbi/create_dbs-04-tst-cdb.json

odacli describe-job -i c0c8b0a0-5033-46b5-81a1-f326f6caa761
...

35 minutes later, my new DB System is ready to use:

Name                  Shape	  GI version          DB info                         Status                  Created                   Updated
--------------------  ----------  ------------------  ------------------------------  ----------------------  ------------------------  ------------------------
dbs-03-tst            dbs2        19.29.0.0.251021    19.29(CONFIGURED=1)             CONFIGURED              2025-12-03 15:05:31 CET   2025-12-08 10:50:06 CET
dbs-04-tst            dbs2        23.26.0.0.0         23.26(CONFIGURED=1)             CONFIGURED              2025-12-08 15:32:48 CET   2025-12-08 16:09:10 CET

Conclusion

Applying this patch is rather easy. Remember these key points when using an ODA:

keep it clean
keep it under control
keep it updated

L’article How to patch your ODA to 19.29? est apparu en premier sur dbi Blog.

You still need reliable and tested backups for your Oracle databases!

Jérôme Dubar — Thu, 16 Oct 2025 12:09:38 +0000

Introduction

These days, I was reviewing and updating the “Oracle Backup and Recovery Workshop” from dbi services. It’s all about RMAN, a powerful tool that always impresses my students. But I must admit that this workshop is less popular than before. Today, backup of an Oracle database is not a hot topic anymore. First, nothing is really new regarding RMAN compared to a few years ago. Hardware reliability has also increased significantly, and data centers are more secured than ever. Both making failures less likely to happen. Another point is that Disaster Recovery has become a standard in most projects, meaning that for a lot of failures, enabling the standby database will solve the problem in a minute or two. Finally, Oracle-embedded flashback technologies bring you faster solutions compared to restoring a backup when database is not broken (these technologies are also part of the workshop). That said, it doesn’t mean that you don’t need a strong backup strategy. This is still your very last protection for dramatic scenarios. And you need to test it from time to time. Let’s review what’s important and how to check and test your backup strategy.

Why you still need a reliable backup at any moment?

Let’s remind the main reasons for having a reliable backup:

you may loose everything on your primary and secondary site
you may have logically corrupted data from a software error or a human error, also replicated to your standby database
you may want to recover partial data that flashback technologies cannot bring back
you may have a security breach widely altering your data (corrupted storage, ransomware, aso)

For sure, this is rather unlikely to happen, but as a DBA, you’re supposed to have a plan in case of any kind of disaster. And a solid RMAN backup is your trustable ally.

Why you probably won’t use a RMAN restore in most cases?

You probably have a Disaster Recovery (DR) setup, meaning 2 sites with database replication using Data Guard (for Enterprise Edition) or Dbvisit Standby (for Standard Edition 2). In most cases, if you need a quick solution for bringing back your database to life after a crash, you will failover to the DR database. It’s a couple of minutes, and if you test your DR database on a regular basis (I mean doing a switchover at least once a year), this operation will succeed without any doubt. A failover is adequate when your primary server is not responding or when your primary database has corrupted files. Doing a failover is always the best thing to do if you want minimum data loss.

If you need data loss, I mean if there is a data corruption due to an error (incorrect SQL commands, application bug), you will probably use the Oracle-embedded flashback technologies. Flashback relies on multiple different technologies for different purposes. If you drop a table, you can bring it back from the recycle bin (FLASHBACK TABLE TO BEFORE DROP), if you delete lines from a table, you can do a query in time (AS OF TIMESTAMP) for example. If the overall database needs to go back in time, and if you’re using Enterprise Edition, you will use FLASHBACK DATABASE. FLASHBACK DATABASE, once enabled, is a mechanism that generates flashback logs, a kind of “INCR -1” automatic backup. With this feature, you can go back at any point in time from a couple of minutes to several hours. It’s much faster than an equivalent RMAN restore. And you can do multiple flashback operations to find the desired Point In Time.

Restoring a backup will be done when nothing else is possible, just because it’s slower to put back older datafiles on disk from backupsets, and recover them.

Probably the most important: make sure RPO and RTO are aligned with the company strategy

As a DBA, you must ensure that these metrics, defined by the company, are met:

Recovery Point Objective: until when you must be able to go back in time regarding data
Recovery Time Objective: the maximum time you need to restore the database to any Point In Time within the RPO

You should consider these RPO/RTO as a contract between the DBA and the management. The DBA will require resources, like adequate infrastructure, disk capacity and licenses to reach these objectives.

Regarding RPO, it’s mainly a matter of storage capacity. For RTO, it involves considering database edition, number of licenses/cores and storage bandwidth.

You must test your backup on a regular basis, just to ensure that you still respect the RTO with an increasing amount of data months after months.

Too often, RPO and RTO are defined by the DBA himself. In other words, the DBA makes what’s possible with the resources he has. This is definitely not the best approach.

Check and test your backups

First, you should never trust your backup! Not because it’s not reliable, but because backup is done when the database is opened, meaning that backups are not consistent. Restore is only possible if a complete set of backups (full + incremental + archivelogs) is available, with an adequate controlfile. Green lights from your backup monitoring tool or the successful word at the end of your backup log is simply not enough. The only trustable backup strategy is the one you test on a regular basis.

These are the checks and tests I would do to make sure my backup is OK.

Check RMAN views

With this statement, you are able to guess the backup strategy and see if it works fine at a glance.

set lines 180
set pages 300
col min for 999.9
col status for a20
alter session set NLS_DATE_FORMAT="DD/MM-HH24:MI";
select start_time "Start", round (input_bytes/1024/1024,1) "Source MB", round(output_bytes/1024/1024) "Backup MB", input_type "Type", status "Status", round(elapsed_seconds/60,1) "Min", round(input_bytes/1024/1024/elapsed_seconds,1) "read MB/s", round(output_bytes/1024/1024/elapsed_seconds,1) "write MB/s", round(compression_ratio,1) RATIO from v$rman_backup_job_details where start_time >= SYSDATE-7 order by 1 desc;

Start           Source MB  Backup MB Type          Status          Min  read MB/s write MB/s      RATIO
-------------- ---------- ---------- ------------- ------------ ------ ---------- ---------- ----------
25/08-15:10:06       1443        380 ARCHIVELOG    COMPLETED        .7       36.1        9.5        3.8
25/08-14:10:06        123         50 ARCHIVELOG    COMPLETED        .1       20.6        8.3        2.5
25/08-13:10:05         33         27 ARCHIVELOG    COMPLETED        .1        6.5        5.4        1.2
25/08-12:10:05         32         27 ARCHIVELOG    COMPLETED        .1          8        6.8        1.2
25/08-11:10:05         46         31 ARCHIVELOG    COMPLETED        .1        7.7        5.1        1.5
25/08-10:10:05         44         31 ARCHIVELOG    COMPLETED        .1        7.4        5.1        1.4
25/08-09:10:04         31         27 ARCHIVELOG    COMPLETED        .1        5.1        4.5        1.1
25/08-08:10:05         32         27 ARCHIVELOG    COMPLETED        .1          8        6.8        1.2
25/08-07:10:05         30         26 ARCHIVELOG    COMPLETED        .1        4.9        4.4        1.1
25/08-06:10:05         30         26 ARCHIVELOG    COMPLETED        .1        5.9        5.3        1.1
25/08-05:10:04         30         27 ARCHIVELOG    COMPLETED        .1          5        4.4        1.1
25/08-04:10:05         32         27 ARCHIVELOG    COMPLETED        .1          8        6.8        1.2
25/08-03:10:05         38         29 ARCHIVELOG    COMPLETED        .1        6.3        4.9        1.3
25/08-02:10:05         30         27 ARCHIVELOG    COMPLETED        .1          5        4.4        1.1
25/08-01:10:05         30         26 ARCHIVELOG    COMPLETED        .1        5.9        5.3        1.1
25/08-00:10:05         39         30 ARCHIVELOG    COMPLETED        .1        7.8          6        1.3
24/08-22:30:06     133858      17371 DB INCR       COMPLETED      26.3       84.8         11        7.7
24/08-22:10:05       1238        376 ARCHIVELOG    COMPLETED        .5       41.3       12.5        3.3
24/08-21:10:05         29         26 ARCHIVELOG    COMPLETED        .1        7.2        6.6        1.1
24/08-20:10:05         29         26 ARCHIVELOG    COMPLETED        .1        4.1        3.8        1.1
24/08-19:10:05         34         28 ARCHIVELOG    COMPLETED        .1        5.6        4.6        1.2
24/08-18:10:05         39         29 ARCHIVELOG    COMPLETED        .1        7.8        5.8        1.3
24/08-17:10:05       1037        204 ARCHIVELOG    COMPLETED        .5       35.8          7        5.1
24/08-16:10:05         42         31 ARCHIVELOG    COMPLETED        .1          6        4.4        1.4
...

In this example, everything is fine (Status=COMPLETED), a full (INCR level 0) is done every night, archivelog backup is done every hour and compression is enabled. Here, read MB/s and write MB/s give you estimated values. If you use incremental backups (INCR level 1), you can identify them with the “Backup MB” column: the figure should be much lower than the one of an INCR level 0.

Having a catalog is definitely more convenient as you can get an overview for all your databases from a unique statement:

set lines 180
set pages 300
col min for 999.9
col status for a20
alter session set NLS_DATE_FORMAT="DD/MM-HH24:MI:SS";
select db_name, start_time "Start", round (input_bytes/1024/1024,1) "Source MB", round(output_bytes/1024/1024,1) "Backup MB", input_type "Type", status "Status", round(elapsed_seconds/60,1) "Min", round(input_bytes/1024/1024/elapsed_seconds,1) "read MB/s", round(output_bytes/1024/1024/elapsed_seconds,1) "write MB/s", round(compression_ratio,1) RATIO from rc_rman_backup_job_details where start_time >= SYSDATE-7  order by 1,2 desc;
...

But this query is not enough to make sure that you can restore your database.

Check if retention is the one you expect

It’s not rare that a DBA is surprised when he checks backup retention and discovers a much lower value than expected. Retention can be changed in a minute, often for solving storage capacity issues. This could have dramatic consequences if you miss the RPO your boss sign up for. But most probably, nobody will know about that…

rman target /
show RETENTION POLICY ;

RMAN configuration parameters for database with db_unique_name ROP_S1 are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;

With this retention, you should be able to restore the database at a consistent state between now and 7 days before. It implies keeping backupsets of database for more than 7 days, because you need the FULL or INCR 0 backup before 7 days to reach the oldest Point In Time. And for sure, all the INCR 1 and archivelog backups done after this FULL or INCR 0.

But checking the retention is not enough to make sure you have everything needed for a restore.

Check if backups are known by RMAN

When deleting backups, this retention can be overridden if needed. So there is no guarantee that backups are still in the catalog. You should be able to find backups within this retention, backups of database (FULL or INCR 0) but also backups of archivelogs and INCR 1:

LIST BACKUP OF DATABASE COMPLETED BETWEEN 'sysdate -7' AND 'sysdate';
...

LIST BACKUP OF ARCHIVELOG COMPLETED BETWEEN 'sysdate -7' AND 'sysdate';
...

This is pretty verbose but you will find the various backusets and what’s inside.

Unfortunately, this is still not enough to make sure you can restore your database.

Do a crosscheck to validate that backup pieces are available

LIST BACKUP is only having a look at backup references in the catalog, it doesn’t guarantee that backupset files (backup pieces) are available. Therefore, a crosscheck is needed to match references with files on disk:

CROSSCHECK BACKUPSET;

You can do crosscheck only on a subset of backups, for example the most recents:

CROSSCHECK BACKUPSET COMPLETED AFTER 'sysdate -1';
...

Or backups between 2 dates:

CROSSCHECK BACKUPSET COMPLETED BETWEEN 'sysdate -7' AND 'sysdate -6';
...

All backupsets must be AVAILABLE. If not, they are EXPIRED, meaning not on disk anymore. This is not a normal behavior.

Is it enough for making sure you can restore? Not at all, it doesn’t mean the restore is possible.

Do a restore database validate with and without a Point In Time

Without any downtime and disk usage, you can do a RESTORE DATABASE VALIDATE. It will read backupsets from the backup location and validate that these backupsets are OK to restore the datafiles.

run {
ALLOCATE CHANNEL C1 DEVICE TYPE DISK;
ALLOCATE CHANNEL C2 DEVICE TYPE DISK;
ALLOCATE CHANNEL C3 DEVICE TYPE DISK;
RESTORE DATABASE VALIDATE;
}

This RESTORE DATABASE VALIDATE is also possible with a Point In Time. RMAN is then able to simulate a full restore of the adequate backupsets within the rentention.

run {
SET UNTIL TIME "TO_DATE('10-OCT-2025 14:00:00','DD-MON-YYYY HH24:MI:SS')";
ALLOCATE CHANNEL C1 DEVICE TYPE DISK;
ALLOCATE CHANNEL C2 DEVICE TYPE DISK;
ALLOCATE CHANNEL C3 DEVICE TYPE DISK;
RESTORE DATABASE VALIDATE;
}

Note that it restores the datafiles to /dev/null, meaning nowhere. This is why a RECOVER DATABASE VALIDATE does not exist, it’s the main limit. You will be able to restore the database, but you might not be able to recover the datafiles and open the database.

This test is nice, but not good enough. The recover is mandatory to open the database.

Do a restore AND recover on another server with or without a Point In Time

The best test would be a complete restore/recover on another server. Considering this server has the same system settings, the same Oracle version and the same filesystems, it’s rather easy. First, identify the latest controlfile and spfile autobackup, then restore the spfile and the controlfile from this backupset. RMAN is able to start an instance without spfile, for the only purpose of restoring the spfile:

su - oracle
echo "BMC3PRD:/u02/app/oracle/product/19.0.0.0/dbhome_1:Y" >> /etc/oratab

ls -lrt /backup/BMC3PRD/ | tail -n 1 
-rw-r-----. 1 oracle oinstall 19136512 Sep  4 09:20 o1_mf_s_1156169494_lr8gd9h4_.bkp

. oraenv <<< BMC3PRD

rman target / 

startup nomount;
startup failed: ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u02/app/oracle/product/19.0.0.0/dbhome_1/dbs/initBMC3PRD.ora'

starting Oracle instance without parameter file for retrieval of spfile
Oracle instance started

Total System Global Area    1070428224 bytes

Fixed Size                     8904768 bytes
Variable Size                281018368 bytes
Database Buffers             771751936 bytes
Redo Buffers                   8753152 bytes


restore spfile from '/backup/BMC3PRD/o1_mf_s_1156169494_lr8gd9h4_.bkp';
Starting restore at 04-SEP-25
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=433 device type=DISK
channel ORA_DISK_1: restoring spfile from AUTOBACKUP /backup/BMC3PRD/o1_mf_s_1156169494_lr8gd9h4_.bkp
channel ORA_DISK_1: SPFILE restore from AUTOBACKUP complete
Finished restore at 04-SEP-25
 
startup force nomount;

restore controlfile from '/backup/BMC3PRD/o1_mf_s_1156169494_lr8gd9h4_.bkp';

Starting restore at 04-SEP-25
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=26 device type=DISK
channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:05
output file name=+RECO/BMC3PRD_CR/CONTROLFILE/current.486.1210937913
Finished restore at 04-SEP-25

startup force mount;
run {
ALLOCATE CHANNEL C1 DEVICE TYPE DISK;
ALLOCATE CHANNEL C2 DEVICE TYPE DISK;
ALLOCATE CHANNEL C3 DEVICE TYPE DISK;
ALLOCATE CHANNEL C4 DEVICE TYPE DISK;
RESTORE DATABASE ;
RECOVER DATABASE ;
ALTER DATABASE OPEN RESETLOGS;
}

...

Testing a restore/recover at a random Point In Time within your retention window is also quite easy: identify the autobackup of spfile and controlfile corresponding to the first one after your Point in Time target, and specify an UNTIL clause in your restore/recover database block:

rman target / 

startup nomount;
startup failed: ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u02/app/oracle/product/19.0.0.0/dbhome_1/dbs/initBMC3PRD.ora'

starting Oracle instance without parameter file for retrieval of spfile
Oracle instance started

Total System Global Area    1070428224 bytes

Fixed Size                     8904768 bytes
Variable Size                281018368 bytes
Database Buffers             771751936 bytes
Redo Buffers                   8753152 bytes


restore spfile from '/backup/BMC3PRD/o1_mf_s_1156169494_124efa11_.bkp';
...

startup force nomount;

restore controlfile from '/backup/BMC3PRD/o1_mf_s_1156169494_124efa11_.bkp';
...

startup force mount;
run {
SET UNTIL TIME "TO_DATE('10-SEP-2025 09:27:42','DD-MON-YYYY HH24:MI:SS')";
ALLOCATE CHANNEL C1 DEVICE TYPE DISK;
ALLOCATE CHANNEL C2 DEVICE TYPE DISK;
ALLOCATE CHANNEL C3 DEVICE TYPE DISK;
ALLOCATE CHANNEL C4 DEVICE TYPE DISK;
RESTORE DATABASE ;
RECOVER DATABASE ;
ALTER DATABASE OPEN RESETLOGS;
}
...

This time, this test is enough to make sure your backups are OK, because you just did the exact same procedure as if you were facing a real disaster case.

Duplicate from backup with or without a Point In Time

You may not want to restore the database with the same name than the one used for production database, for obvious reasons. You can then restore with a different name thanks to the DUPLICATE command. This command is used for duplicating a target (source) to an auxiliary database (destination), but you can use it without any target, only pointing to a folder containing the backupsets. Starting the instance without a spfile is still possible with RMAN, but you’ll need to connect using target keyword first:

echo "RESTO:/u02/app/oracle/product/19.0.0.0/dbhome_1:Y" >> /etc/oratab

. oraenv <<< RESTO

rman target /
startup nomount;
exit;

rman auxiliary /
run {
ALLOCATE AUXILIARY CHANNEL C1 DEVICE TYPE DISK;
ALLOCATE AUXILIARY CHANNEL C2 DEVICE TYPE DISK;
ALLOCATE AUXILIARY CHANNEL C3 DEVICE TYPE DISK;
ALLOCATE AUXILIARY CHANNEL C4 DEVICE TYPE DISK;
DUPLICATE DATABASE TO RESTO BACKUP LOCATION '/backup/BMC3PRD/' NOFILENAMECHECK;
}

It also works with a Point In Time:

rman auxiliary /
run {
SET UNTIL TIME "TO_DATE('10-OCT-2025 16:30:00','DD-MON-YYYY HH24:MI:SS')";
ALLOCATE AUXILIARY CHANNEL C1 DEVICE TYPE DISK;
ALLOCATE AUXILIARY CHANNEL C2 DEVICE TYPE DISK;
ALLOCATE AUXILIARY CHANNEL C3 DEVICE TYPE DISK;
ALLOCATE AUXILIARY CHANNEL C4 DEVICE TYPE DISK;
DUPLICATE DATABASE TO RESTO BACKUP LOCATION '/backup/BMC3PRD/' NOFILENAMECHECK;
}

All these commands are fine when using OMF: I would recommend using OMF everywhere. If you don’t use OMF, you will need to take care of file names and use path conversions within RMAN commands.

Please, never do a DUPLICATE on a server where a database with the same name as the source database exists: at some point, RMAN needs to restore the controlfile with the name of the source database before changing for the new name, it can be hazardous…

This test is as good as the previous one: it can definitely validate your backups as if you were restoring your production database.

Conclusion

I would summarize this blog post with these 3 things:

you still need a strong backup strategy: do not underestimate the high importance your backups still have
RPO and RTO are decided by your company and the DBA asks for resources allowing him to reach these targets. He must warn if resources cannot guarantee these RPO/RTO
do regular and extensive checks of your backups, and simulate a disaster scenario at least once a year: this is good for validating RPO/RTO and good for validating the reliability of your backup strategy. This is also good for your training as you normally won’t do regular restore/recover operations

L’article You still need reliable and tested backups for your Oracle databases! est apparu en premier sur dbi Blog.

Cannot drop a tablespace?

Jérôme Dubar — Fri, 26 Sep 2025 13:43:42 +0000

Introduction

Removing a tablespace shouldn’t be a problem as soon as no segments remain in it. So why sometimes Oracle prevents you from removing a tablespace?

My database

The database I’m working on is a 19c database for SAP. A previously used tablespace has been put offline several months ago, and it’s now time to remove it from the database. Easy task.

Prerequisites before removal

Let’s first identify the tablespace:

. oraenv <<< DOT
sqlplus / as sysdba

select tablespace_name, status from dba_tablespaces where status = 'OFFLINE';
TABLESPACE_NAME 	       STATUS
------------------------------ ---------
PSAPSR3740		       OFFLINE

Now, let’s check if there are segments in this tablespace:

col owner for a20
col segment_type for a30
col segment_name for a50
set lines 200
set pages 200
select owner, segment_type, segment_name from dba_segments where tablespace_name='PSAPSR3740'; 
OWNER		     SEGMENT_TYPE		    SEGMENT_NAME
-------------------- ------------------------------ --------------------------------------------------
SAPSR3		     TEMPORARY			    7.130
SAPSR3		     TEMPORARY			    7.138
SAPSR3		     TEMPORARY			    7.146
SAPSR3		     TEMPORARY			    7.154
SAPSR3		     TEMPORARY			    7.162
SAPSR3		     TEMPORARY			    7.170
SAPSR3		     TEMPORARY			    7.178
SAPSR3		     TEMPORARY			    7.186
SAPSR3		     TEMPORARY			    7.194
SAPSR3		     TEMPORARY			    7.210
SAPSR3		     TEMPORARY			    7.218
SAPSR3		     TEMPORARY			    7.226
SAPSR3		     TEMPORARY			    7.234
SAPSR3		     TEMPORARY			    7.242
SAPSR3		     TEMPORARY			    7.250
SAPSR3		     TEMPORARY			    7.1027610
…
SAPSR3		     TEMPORARY			    12.1317458
SAPSR3		     TEMPORARY			    12.1338026
SAPSR3		     TEMPORARY			    12.1338082
SAPSR3		     TEMPORARY			    12.1338090
SAPSR3		     TEMPORARY			    12.1338098
SAPSR3		     TEMPORARY			    12.1382362
SAPSR3		     TEMPORARY			    12.1384850
SAPSR3		     TEMPORARY			    12.1384898
SAPSR3		     TEMPORARY			    12.1384922
SAPSR3		     TEMPORARY			    12.1384930
SAPSR3		     TEMPORARY			    12.1384938
SAPSR3		     TEMPORARY			    12.1386154

Temporary segments are located in this tablespace. Let’s put back this tablespace online, and then remove these temporary segments with COALESCE.

ALTER TABLESPACE PSAPSR3740 online;
Tablespace altered.

ALTER TABLESPACE PSAPSR3740 coalesce;
Tablespace altered.

select owner, segment_type, segment_name from dba_segments where tablespace_name='PSAPSR3740';
no rows selected

This tablespace could also be the default’s one for some users, let’s check that:

select username from dba_users where default_tablespace='PSAPSR3740';
no rows selected

This tablespace could normally be removed now.

Error when removing the tablespace

Let’s now try to drop this tablespace:

DROP TABLESPACE PSAPSR3740 INCLUDING CONTENTS AND DATAFILES;
DROP TABLESPACE PSAPSR3740 INCLUDING CONTENTS AND DATAFILES
*
ERROR at line 1:
ORA-22868: table with LOBs contains segments in different tablespaces

There is still an object linked to this tablespace, let’s try to identify it:

select * from v$tablespace where name='PSAPSR3740';
       TS# NAME 		     INC BIG FLA ENC	 CON_ID
---------- ------------------------- --- --- --- --- ----------
	 9 PSAPSR3740		     YES NO  YES	      0

select t.obj# from tab$ t,ind$ i where i.type#=8 and i.ts#=9 and i.bo#=t.obj# and bitand(t.property, 32+64)=0 and t.ts#<>9 union all select null from tab$ t,ind$ iot,ind$ i where i.type#=8 and i.ts#=9 and i.bo#=t.obj# and bitand(t.property, 32)=0 and bitand(t.property, 64)<>0 and t.pctused$=iot.obj# and iot.ts#<>9 ;
      OBJ#
----------
    173459

col object_name for a30
select owner, object_name, object_type from dba_objects where object_id=173459;
OWNER		     OBJECT_NAME		    OBJECT_TYPE
-------------------- ------------------------------ -----------------------
SAPSR3		     DBMAPS			    TABLE

Let’s check if this object has a segment, meaning that it needs tablespace storage:

select tablespace_name from dba_segments where segment_name='DBMAPS';
no rows selected

select count(*) from SAPSR3.DBMAPS;
  COUNT(*)
----------
	 0

The table is empty, and as Enterprise Edition has deferred segment creation, the segment doesn’t exist at all.

Let’s have a look at the DDL of this object:

set long 200000 pages 0 lines 131
column txt format a121 word_wrapped
SELECT DBMS_METADATA.GET_DDL('TABLE','DBMAPS','SAPSR3') from dual;
  CREATE TABLE "SAPSR3"."DBMAPS"
   (	"TABNAME" VARCHAR2(30) DEFAULT ' ' NOT NULL ENABLE,
	"VARKEY" VARCHAR2(180) DEFAULT ' ' NOT NULL ENABLE,
	"DATALN" NUMBER(5,0) DEFAULT 0 NOT NULL ENABLE,
	"VARDATA" BLOB
   ) SEGMENT CREATION DEFERRED
  PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE( INITIAL 16384 NEXT 163840 MINEXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0)
  TABLESPACE "PSAPSR3750"
 LOB ("VARDATA") STORE AS SECUREFILE (
  TABLESPACE "PSAPSR3740" ENABLE STORAGE IN ROW CHUNK 8192
  CACHE  NOCOMPRESS  KEEP_DUPLICATES );

This table has a LOB column, and the associated storage clause for this LOB points to the tablespace I want to delete. Let’s change the table definition and use another online tablespace for the storage clause:

ALTER TABLE SAPSR3.DBMAPS MOVE TABLESPACE PSAPSR3750
LOB ("VARDATA") STORE AS SECUREFILE (
  TABLESPACE "PSAPSR3750" ENABLE STORAGE IN ROW CHUNK 8192
  CACHE  NOCOMPRESS  KEEP_DUPLICATES );
Table altered.

Now let’s try again to remove the tablespace:

DROP TABLESPACE PSAPSR3740 INCLUDING CONTENTS AND DATAFILES;
Tablespace dropped.

It’s done now.

Conclusion

The storage clause for a LOB was preventing the tablespace from being removed, although no storage was used.

L’article Cannot drop a tablespace? est apparu en premier sur dbi Blog.

Reimage your ODA from scratch

Jérôme Dubar — Tue, 09 Sep 2025 17:45:04 +0000

Introduction

Oracle Database Appliance is still very popular today. Provisioning is fast: from unboxing to a first running database is done in one day. And everything is standardized.

Patching is not that straight forward. First, you can only apply the latest patch on top of the 4 previous ones, and if you come from an older release, you may have 2, 3 or more patches to apply to reach the current version. It could take hours, even days.

The question one could ask is if it wouldn’t be more efficient to do a full ODA reimaging instead of patching. Let’s try to find out.

Environment and upgrade plan

This is a “real” environment composed of 4x ODA X8-2M: 2x ODAs for production, 2x ODAs for test/development. These servers have been provisioned in 2020 with the first 19c production available on ODA: 19.6. They were updated twice in the past years to reach 19.20 in 2024. These 2 upgrades were not that easy for some reasons, and systems are not that clean in my opinion.

According to MOS note 2757884.1, these ODAs are supported until August 2027, meaning that the last patch for these servers will be provided at this time. My client wants to keep them for 2 more years at least. So it’s definitely worth it to apply the patch.

I could have applied patch 19.24 then patch 19.27 on these ODAs, but as the first jump requires a Data Preserving Reprovisioning (meaning an OS reimaging) and as my ODAs are not that clean, I discussed with my client and we chose to reimage the servers from scratch.

Note that the production ODAs we will focus on are running 15 Enterprise Edition databases, and that a Data Guard configuration exists for each of them.

The plan is to switch all primaries to one ODA, reimage the other one, wait 2 weeks to see if everything is fine, do the switchover to the freshly reimaged ODA and reimage the other one the same way. As I don’t want to patch my databases now, I will deploy a 19.20 DB home and migrate to the latest version later.

Note that when I started this reimaging, current patch was 19.27. In the meantime, 19.28 becomes available.

Before starting: check ODA’s health

Let’s first check if the ODA I planned to reimage is healthy. You should normally monitor the hardware alerts on your ODAs, and eventually configure Automatic Service Request (ASR) to address hardware issues as soon as possible.

odaadmcli show server | grep Problem
	Open Problems            : 0
	Open Problems Report     : System is healthy

No problem on this ODA. I wouldn’t do anything on an ODA that is not healthy: fix up hardware troubles before patching or reimaging an ODA.

Check if standby databases are synced and disable archive transport

Apart from DBTEST which is primary (I always have a standalone test database on my ODAs), every database must be a Physical Standby Database. Expected downtime for such a reimaging is several hours or days depending on the number of databases. So, checking if sync is OK is a good practice. I also disable transport to standby databases and step down Data Guard to Maximum Performance:

vi check_role.sql
conn / as sysdba
set pages 0
select name||' is here '|| database_role from v$database;
exit;

for a in `ps -ef | grep pmon  | grep -v grep | grep -v ASM | grep -v APX | cut -c 58- | sort`; do  . oraenv <<< $a 1> /dev/null ; sqlplus -s /nolog @check_role; done
BPAPRD is here PHYSICAL STANDBY

DBTEST is here PRIMARY

BMC3PRD is here PHYSICAL STANDBY

MLOGAR is here PHYSICAL STANDBY

NIMSHMP is here PHYSICAL STANDBY

NOVOTEC is here PHYSICAL STANDBY

NTRAMPRD is here PHYSICAL STANDBY

NWTASKP is here PHYSICAL STANDBY

OPERA is here PHYSICAL STANDBY

PBYPRD is here PHYSICAL STANDBY

PRAXPRD is here PHYSICAL STANDBY

RCAT is here PHYSICAL STANDBY

SMARTPRD is here PHYSICAL STANDBY

SMKPRD is here PHYSICAL STANDBY

SPORT1 is here PHYSICAL STANDBY

SPORT2 is here PHYSICAL STANDBY


for a in `ps -ef | grep pmon  | grep -v grep | grep -v ASM | grep -v APX | grep -v DBTEST | cut -c 58- | sort`; do echo $a ; . oraenv <<< $a 1> /dev/null ; dgmgrl / "show configuration lag" | grep Lag; done
BPAPRD
             Transport Lag:	 0 seconds (computed 1 second ago)
             Apply Lag:          0 seconds (computed 1 second ago)
BMC3PRD
                  Transport Lag:      0 seconds (computed 0 seconds ago)
                  Apply Lag:          0 seconds (computed 0 seconds ago)
MLOGAR
                   Transport Lag:      0 seconds (computed 1 second ago)
                   Apply Lag:          0 seconds (computed 1 second ago)
NIMSHMP
                  Transport Lag:      0 seconds (computed 0 seconds ago)
                  Apply Lag:          0 seconds (computed 0 seconds ago)
NOVOTEC
                  Transport Lag:      0 seconds (computed 0 seconds ago)
                  Apply Lag:          0 seconds (computed 0 seconds ago)
NTRAMPRD
                    Transport Lag:	0 seconds (computed 0 seconds ago)
                    Apply Lag:          0 seconds (computed 0 seconds ago)
NWTASKP
                  Transport Lag:      0 seconds (computed 0 seconds ago)
                  Apply Lag:          0 seconds (computed 0 seconds ago)
OPERA
             Transport Lag:	 0 seconds (computed 0 seconds ago)
             Apply Lag:          0 seconds (computed 0 seconds ago)
PBYPRD
                  Transport Lag:      0 seconds (computed 0 seconds ago)
                  Apply Lag:          0 seconds (computed 0 seconds ago)
PRAXPRD
             Transport Lag:	 0 seconds (computed 1 second ago)
             Apply Lag:          0 seconds (computed 1 second ago)
RCAT
              Transport Lag:	  0 seconds (computed 1 second ago)
              Apply Lag:          0 seconds (computed 1 second ago)
 
SMARTPRD
             Transport Lag:	 0 seconds (computed 0 seconds ago)
             Apply Lag:          0 seconds (computed 0 seconds ago)
SMKPRD
             Transport Lag:	 0 seconds (computed 1 second ago)
             Apply Lag:          0 seconds (computed 1 second ago)
SPORT1
             Transport Lag:	 0 seconds (computed 0 seconds ago)
             Apply Lag:          0 seconds (computed 0 seconds ago)
SPORT2
             Transport Lag:	 0 seconds (computed 0 seconds ago)
             Apply Lag:          0 seconds (computed 0 seconds ago)

for a in `ps -ef | grep pmon  | grep -v grep | grep -v ASM | grep -v APX | cut -c 58- | sort`; do echo $a ; . oraenv <<< $a 1> /dev/null ; dgmgrl / "edit configuration set protection mode as MaxPerformance;" | grep Succeeded; dgmgrl / "EDIT DATABASE ${a}_IN SET STATE='TRANSPORT-OFF';" | grep Succeed ; done
BPAPRD
Succeeded.
Succeeded.
DBTEST
BMC3PRD
Succeeded.
Succeeded.
MLOGAR
Succeeded.
Succeeded
NIMSHMP
Succeeded.
Succeeded.
NOVOTEC
Succeeded.
Succeeded.
NTRAMPRD
Succeeded.
Succeeded
NWTASKP
Succeeded.
Succeeded.
OPERA
Succeeded.
Succeeded.
PBYPRD
Succeeded.
Succeeded.
PRAXPRD
Succeeded.
Succeeded.
RCAT
Succeeded.
Succeeded.
SMARTPRD
Succeeded.
Succeeded.
SMKPRD
Succeeded.
Succeeded.
SPORT1
Succeeded.
Succeeded.
SPORT2
Succeeded.
Succeeded.

Backup important files

As you will erase everything on this ODA, backup everything that needs to be backed up on a remote filesystem (/backup here). Usually, I use this script:

cat /backup/Patch19.27/dbi_prepatch_backup.sh
# Backup important files before patching
export BKPPATH=/backup/Patch19.27/backup_ODA_`hostname`_`date +"%Y%m%d_%H%M"`
echo "Backing up to " $BKPPATH
mkdir -p $BKPPATH
odacli list-databases > $BKPPATH/list-databases.txt
ps -ef | grep pmon | grep -v ASM | grep -v APX | grep -v grep | cut -c 58- | sort > $BKPPATH/running-instances.txt
odacli list-dbhomes > $BKPPATH/list-dbhomes.txt
odacli list-dbsystems > $BKPPATH/list-dbsystems.txt
odacli list-vms > $BKPPATH/list-vms.txt
crontab -u oracle -l  > $BKPPATH/crontab-oracle.txt
crontab -u grid -l  > $BKPPATH/crontab-grid.txt
crontab -l  > $BKPPATH/crontab-root.txt

cat /etc/fstab >  $BKPPATH/fstab.txt
cat /etc/oratab >  $BKPPATH/oratab.txt
cat /etc/sysconfig/network >  $BKPPATH/etc-sysconfig-network.txt
cat /etc/hosts  >  $BKPPATH/hosts
cat /etc/resolv.conf  >  $BKPPATH/resolv.conf
cat /etc/sysctl.conf  >  $BKPPATH/

cp /etc/krb5.conf  $BKPPATH/
cp /etc/krb5.keytab  $BKPPATH/
mkdir $BKPPATH/network-scripts
cp  /etc/sysconfig/network-scripts/ifcfg*  $BKPPATH/network-scripts/
odacli describe-system > $BKPPATH/describe-system.txt
odacli  describe-component >  $BKPPATH/describe-component.txt
HISTFILE=~/.bash_history
set -o history
history > $BKPPATH/history-root.txt
cp /home/oracle/.bash_history $BKPPATH/history-oracle.txt
df -h >  $BKPPATH/filesystems-status.txt

for a in `odacli list-dbhomes -j | grep dbHomeLocation | awk -F '"' '{print $4}' | sort` ; do mkdir -p $BKPPATH/$a/network/admin/ ; cp $a/network/admin/tnsnames.ora $BKPPATH/$a/network/admin/; cp $a/network/admin/sqlnet.ora $BKPPATH/$a/network/admin/; done
for a in `odacli list-dbhomes -j | grep dbHomeLocation | awk -F '"' '{print $4}' | sort` ; do mkdir -p $BKPPATH/$a/owm/ ; cp -r $a/owm/* $BKPPATH/$a/owm/; done
cp `ps -ef | grep -v grep | grep LISTENER | awk -F ' ' '{print $8}' | awk -F 'bin' '{print $1}'`network/admin/listener.ora $BKPPATH/gridhome-listener.ora
cp `ps -ef | grep -v grep | grep LISTENER | awk -F ' ' '{print $8}' | awk -F 'bin' '{print $1}'`/network/admin/sqlnet.ora $BKPPATH/gridhome-sqlnet.ora

tar czf $BKPPATH/u01-app-oracle-admin.tgz /u01/app/oracle/admin/
tar czf $BKPPATH/u01-app-odaorabase-oracle-admin.tgz /u01/app/odaorabase/oracle/admin/
tar czf $BKPPATH/u01-app-oracle-local.tgz /u01/app/oracle/local/
tar czf $BKPPATH/home.tgz /home/
cp /etc/passwd $BKPPATH/
cp /etc/group $BKPPATH/

echo "End"
echo "Backup files size:"
du -hs  $BKPPATH
echo "Backup files content:"
ls -lrt  $BKPPATH

sh /backup/Patch19.27/dbi_prepatch_backup.sh
...

This backup script is probably not extensive: take some time to make a suitable one for your needs.

Let’s also take a backup of the DB homes, in my case I only have one:

tar czf /backup/Patch19.27/odaprdcr_dbhome.tgz /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4

ls -lrth /backup/Patch19.27/odaprdcr_dbhome.tgz

-rw-r--r-- 1 root root 5.9G Sep  3 10:50 /backup/Patch19.27/odaprdcr_dbhome.tgz

Backup standby spfiles and controlfiles

As this ODA is only running standby databases, I don’t need to back them up. I can use the backup from primary assuming a reliable backup strategy has been deployed on production’s databases. 2 files differ between primary and standby databases: spfile and controlfile. So let’s backup these 2 files on all databases, it just needs a couple of minutes for all:

su - oracle

mkdir /backup/Patch19.27/ctl_spf

cat /backup/Patch19.27/bck_ctl_spf.rcv

backup current controlfile format '/backup/Patch19.27/ctl_spf/ctl_%d_%T.bck';
backup spfile format '/backup/Patch19.27/ctl_spf/spf_%d_%T.bck';
exit;

for a in `ps -ef | grep pmon  | grep -v grep | grep -v ASM | grep -v APX | grep -v DBTEST | cut -c 58- | sort`; do echo $a ; . oraenv <<< $a 1> /dev/null ; rman target / cmdfile="/backup/Patch19.27/bck_ctl_spf.rcv" | grep Succeeded; done

ls -lrt /backup/Patch19.27/ctl_spf

total 33592
-rw-r----- 1 oracle asmadmin 46104576 Sep  3 10:33 ctl_BPAPRD_20250903.bck
-rw-r----- 1 oracle asmadmin   114688 Sep  3 10:33 spf_BPAPRD_20250903.bck
-rw-r----- 1 oracle asmadmin 21233664 Sep  3 10:33 ctl_BMC3PRD_20250903.bck
-rw-r----- 1 oracle asmadmin   114688 Sep  3 10:33 spf_BMC3PRD_20250903.bck
-rw-r----- 1 oracle asmadmin 27656192 Sep  3 10:33 ctl_MLOGAR_20250903.bck
-rw-r----- 1 oracle asmadmin   114688 Sep  3 10:34 spf_MLOGAR_20250903.bck
-rw-r----- 1 oracle asmadmin 30965760 Sep  3 10:34 ctl_NIMSHMP_20250903.bck
-rw-r----- 1 oracle asmadmin   114688 Sep  3 10:34 spf_NIMSHMP_20250903.bck
-rw-r----- 1 oracle asmadmin 27394048 Sep  3 10:34 ctl_NOVOTEC_20250903.bck
-rw-r----- 1 oracle asmadmin   114688 Sep  3 10:34 spf_NOVOTEC_20250903.bck
-rw-r----- 1 oracle asmadmin 77955072 Sep  3 10:34 ctl_NTRAMPRD_20250903.bck
-rw-r----- 1 oracle asmadmin   114688 Sep  3 10:34 spf_NTRAMPRD_20250903.bck
-rw-r----- 1 oracle asmadmin 29294592 Sep  3 10:34 ctl_NWTASKP_20250903.bck
-rw-r----- 1 oracle asmadmin   114688 Sep  3 10:34 spf_NWTASKP_20250903.bck
-rw-r----- 1 oracle asmadmin 66387968 Sep  3 10:35 ctl_OPERA_20250903.bck
-rw-r----- 1 oracle asmadmin   114688 Sep  3 10:35 spf_OPERA_20250903.bck
-rw-r----- 1 oracle asmadmin 20905984 Sep  3 10:35 ctl_PBYPRD_20250903.bck
-rw-r----- 1 oracle asmadmin   114688 Sep  3 10:35 spf_PBYPRD_20250903.bck
-rw-r----- 1 oracle asmadmin 28868608 Sep  3 10:35 ctl_PRAXPRD_20250903.bck
-rw-r----- 1 oracle asmadmin   114688 Sep  3 10:35 spf_PRAXPRD_20250903.bck
-rw-r----- 1 oracle asmadmin 14352384 Sep  3 10:35 ctl_RCAT_20250903.bck
-rw-r----- 1 oracle asmadmin   114688 Sep  3 10:35 spf_RCAT_20250903.bck
-rw-r----- 1 oracle asmadmin 35028992 Sep  3 10:35 ctl_SMARTPRD_20250903.bck
-rw-r----- 1 oracle asmadmin   114688 Sep  3 10:35 spf_SMARTPRD_20250903.bck
-rw-r----- 1 oracle asmadmin 21594112 Sep  3 10:35 ctl_SMKPRD_20250903.bck
-rw-r----- 1 oracle asmadmin   114688 Sep  3 10:36 spf_SMKPRD_20250903.bck
-rw-r----- 1 oracle asmadmin 36798464 Sep  3 10:36 ctl_SPORT1_20250903.bck
-rw-r----- 1 oracle asmadmin   114688 Sep  3 10:36 spf_SPORT1_20250903.bck
-rw-r----- 1 oracle asmadmin 20840448 Sep  3 10:36 ctl_SPORT2_20250903.bck
-rw-r----- 1 oracle asmadmin   114688 Sep  3 10:36 spf_SPORT2_20250903.bck

The restore will be easier with these files.

Prepare the json file for ODA deployment

Before doing the cleanup, prepare the json file you will use for deploying again your ODA.

You can use the one used for initial deployment, or eventually create a new one.

You will need these things for the json:

which edition you use: Standard Edition or Enterprise Edition
hostname, IP, netmask and gateway of your server
hostname, IP, netmask and gateway of your ILOM
user and group ids if you don’t use the default ones
ratio between DATA and RECO for disk partitioning (you may change from what was configured before)
DNS and NTP IPs
default’s network interface (usually btbond1)
DBTEST parameters if you create a test database during provisioning (I would recommend doing so)

You will find most of these parameters in the backup you’ve done with the script.

Here is my json file:

{
    "instance": {
        "instanceBaseName": "odaprdcr",
        "dbEdition": "EE",
        "objectStoreCredentials": null,
        "name": "odaprdcr",
        "systemPassword": "*********",
        "timeZone": "Europe/Zurich",
        "domainName": "dbi-services.ch",
        "ntpServers" : ["10.8.122.132","10.8.122.133"],
        "dnsServers" : ["10.8.122.132","10.8.122.233"],
        "isMultiUserAccessEnabled": "false",
        "isRoleSeparated": true,
        "osUserGroup": {
            "users": [
                {
                    "userName": "oracle",
                    "userRole": "oracleUser",
                    "userId": 1000
                },
                {
                    "userName": "grid",
                    "userRole": "gridUser",
                    "userId": 1001
                }
            ],
            "groups": [
                {
                    "groupName": "oinstall",
                    "groupRole": "oinstall",
                    "groupId": 1001
                },
                {
                    "groupName": "dbaoper",
                    "groupRole": "dbaoper",
                    "groupId": 1002
                },
                {
                    "groupName": "dba",
                    "groupRole": "dba",
                    "groupId": 1000
                },
                {
                    "groupName": "asmadmin",
                    "groupRole": "asmadmin",
                    "groupId": 1004
                },
                {
                    "groupName": "asmoper",
                    "groupRole": "asmoper",
                    "groupId": 1005
                },
                {
                    "groupName": "asmdba",
                    "groupRole": "asmdba",
                    "groupId": 1006
                }
            ]
        }
    },
    "nodes": [
        {
            "nodeNumber": "0",
            "nodeName": "odaprdcr",
            "network": [
                {
                    "ipAddress" : "10.192.100.103",
                    "subNetMask" : "255.255.255.0",
                    "gateway" : "10.192.100.254",
                    "nicName": "btbond1",
                    "networkType": [
                        "Public"
                    ],
                    "isDefaultNetwork": true
                }
            ],
            "ilom": {
                "ilomName": "odaprdcr-ilom",
                "ipAddress":"10.192.200.103",
                "subNetMask":"255.255.255.0",
                "gateway":"10.192.200.254"
            }
        }
    ],
    "grid": {
        "vip": [],
        "diskGroup": [
            {
                "diskGroupName": "DATA",
                "diskPercentage": 85,
                "redundancy": "NORMAL"
            },
            {
                "diskGroupName": "RECO",
                "diskPercentage": 15,
                "redundancy": "NORMAL"
            }
        ],
        "language": "en",
        "scan": null
    },
    "database": {
        "dbName": "DBTEST",
        "dbCharacterSet": {
            "characterSet": "AL32UTF8",
            "nlsCharacterset": "AL16UTF16",
            "dbTerritory": "AMERICA",
            "dbLanguage": "AMERICAN"
        },
        "dbRedundancy": null,
        "dbRole": "PRIMARY",
        "adminPassword": "*********",
        "dbEdition": "EE",
        "databaseUniqueName": "DBTEST",
        "dbClass": "OLTP",
        "dbVersion": "19.27.0.0.250415",
        "dbHomeId": null,
        "instanceOnly": false,
        "isCdb": true,
        "pdBName": "PDB1",
        "dbShape": "odb1",
        "pdbAdminuserName": "pdbadmin",
        "enableTDE": false,
        "dbType": "SI",
        "dbStorage": "ASM",
        "dbConsoleEnable": false,
        "dbOnFlashStorage": false,
        "backupConfigId": null,
        "rmanBkupPassword": null,
        "tdePassword": null
    },
    "acr": {
        "acrEnable": false
    },
    "sysOraHomeStorage": {
        "diskgroup": "DATA",
        "volumeSize": 80
    }
}

Cleanup the data disks

Cleanup of the data disk headers is mandatory before reimaging, otherwise provisioning will fail. Make sure you’re on the correct ODA, you can’t undo this action:

/opt/oracle/oak/onecmd/cleanup.pl
INFO: Log file is /opt/oracle/oak/log/odaprdcr/cleanup/cleanup_2025-09-03_10-53-04.log
INFO: Log file is /opt/oracle/oak/log/odaprdcr/cleanup/dcsemu_diag_precleanup_2025-09-03_10-53-04.log

INFO: *******************************************************************
INFO: ** Starting process to cleanup provisioned host odaprdcr         **
INFO: *******************************************************************
INFO: Default mode being used to cleanup a provisioned system.
INFO: It will change all ASM disk status from MEMBER to FORMER
Do you want to continue (yes/no) : yes
INFO: nodes will be rebooted
Do you want to continue (yes/no) : yes
INFO: /u01/app/19.20.0.0/oracle/bin/crsctl.bin

INFO: *************************************
INFO: ** Checking for GI bits presence
INFO: *************************************
INFO: GI bits /u01/app/19.20.0.0/oracle found on system under /u01/app directory...

INFO: ** Disabling AFD filtering
SUCCESS: AFD filtering disabled on all devices
INFO: ** Saving disk partition info
INFO: *************************************
INFO: ** Executing Stop Clusterware commands
INFO: *************************************
INFO: *************************************
INFO: ** Remove Oracle Stack
…

INFO: Rebooting the system via ...
INFO: Executing 

INFO: Cleanup was successful
INFO: Log file is /opt/oracle/oak/log/odaprdcr/cleanup/cleanup_2025-09-03_10-53-04.log

WARNING: After system reboot, please re-run "odacli update-repository" for GI/DB clones,
WARNING: before running "odacli create-appliance".

Reimage with the latest OS

ODA patch file 30403643 includes an ISO image you will use to reimage your ODA. From the ILOM, start the remote console, connect this ISO file, define next boot device as CDROM and do a power cycle of the server. Reimaging an ODA takes about 50 minutes and doesn’t need any input. Keep the remote console open for the next operation.

Configure firstnet

Once your ODA runs on a brand new OS, you will need to do the initial network setup through the remote console. So log into the console with root/welcome1, type configure-firstnet and provide IP address, netmask and gateway of your server. LACP mode will depend on your switch configuration, if you didn’t use it before, keep it disabled. On an ODA deployed in 2020, LACP mode was not available. You normally won’t use DHCP, neither VLAN as VLAN is usually tagged on the switch ports where your ODA is connected to.

configure-firstnet
...

Once configured, you can reach your server through the network and copy the needed files for the next steps.

Patch the microcodes

Reimaging your ODA doesn’t mean your microcodes are updated, it’s only the software part that is new. Therefore, first task after reimaging is patching these microcodes.

Let’s copy and register the patch file on the ODA and apply the updates (patch 37817290 for 19.27):

cd /opt/dbi
unzip p37817290_1927000_Linux-x86-64.zip
odacli update-repository -f /opt/dbi/oda-sm-19.27.0.0.0-250601-server.zip

odacli describe-job -i "1e2dd483-2237-47b9-9c60-5395fa720fd2"
Job details
----------------------------------------------------------------
                     ID:  1e2dd483-2237-47b9-9c60-5395fa720fd2
            Description:  Repository Update
                 Status:  Success
                Created:  September 03, 2025 09:53:24 UTC
                Message:  /opt/dbi/oda-sm-19.27.0.0.0-250601-server.zip

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- -------
Unzip bundle                             September 03, 2025 09:53:39 UTC          September 03, 2025 09:53:54 UTC          Success

odacli describe-component
System Version
--------------
19.27.0.0.0

System Node Name
----------------
oak

Local System Version
--------------------
19.27.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                      19.27.0.0.0          up-to-date

DCSCONTROLLER                            19.27.0.0.0          up-to-date

DCSCLI                                   19.27.0.0.0          up-to-date

DCSAGENT                                 19.27.0.0.0          up-to-date

DCSADMIN                                 19.27.0.0.0          up-to-date

OS                                       8.10                 up-to-date

ILOM                                     5.1.1.23.r151750     5.1.4.25.r160118

BIOS                                     52100600             52140100

LOCAL CONTROLLER FIRMWARE {
     [c8, c9]                            214.2.271.9          up-to-date
}

SHARED CONTROLLER FIRMWARE {
     [c0, c1, c2, c3, c4, c5]            VDV1RL06             up-to-date
}

LOCAL DISK FIRMWARE {
     [c6d0, c6d1]                        XC311102             up-to-date
}

HMP                                      2.4.10.1.600         up-to-date

odacli create-prepatchreport -s -v 19.27.0.0.0
 
sleep 120 ; odacli describe-prepatchreport -i 8010302b-feca-4f23-92bc-6878aa586713
Prepatch Report
------------------------------------------------------------------------
                 Job ID:  8010302b-feca-4f23-92bc-6878aa586713
            Description:  Patch pre-checks for [OS, ILOM, SERVER] to 19.27.0.0.0
                 Status:  SUCCESS
                Created:  September 3, 2025 9:54:56 AM UTC
                 Result:  All pre-checks succeeded

Node Name
---------------
oak

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.27.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   There are no packages available for
                                          an update
Validate command execution      Success   Skipped command execution verfication
                                          - Instance is not provisioned

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.27.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Skipped command execution verfication
                                          - Instance is not provisioned

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate command execution      Success   Skipped command execution verfication
                                          - Instance is not provisioned

odacli update-server -v 19.27.0.0.0

sleep 900 ; odacli describe-job -i 3a4ce5a7-6305-483d-a7c3-e6328d8a831e
Job details
----------------------------------------------------------------
                     ID:  3a4ce5a7-6305-483d-a7c3-e6328d8a831e
            Description:  Server Patching to 19.27.0.0.0
                 Status:  Success
                Created:  September 03, 2025 09:57:34 UTC
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- -------
Validate ILOM server reachable           September 03, 2025 09:57:36 UTC          September 03, 2025 09:57:37 UTC          Success
Stop DCS Admin                           September 03, 2025 09:57:38 UTC          September 03, 2025 09:57:39 UTC          Success
Generate mTLS certificates               September 03, 2025 09:57:39 UTC          September 03, 2025 09:57:40 UTC          Success
Exporting Public Keys                    September 03, 2025 09:57:40 UTC          September 03, 2025 09:57:41 UTC          Success
Creating Trust Store                     September 03, 2025 09:57:41 UTC          September 03, 2025 09:57:43 UTC          Success
Restart DCS Admin                        September 03, 2025 09:57:43 UTC          September 03, 2025 09:58:03 UTC          Success
Update config files                      September 03, 2025 09:57:43 UTC          September 03, 2025 09:57:43 UTC          Success
Deactivate Unit[dnf-makecache.timer]     September 03, 2025 09:58:04 UTC          September 03, 2025 09:58:04 UTC          Success
Deactivate Unit[kdump.service]           September 03, 2025 09:58:04 UTC          September 03, 2025 09:58:05 UTC          Success
Modify BM udev rules                     September 03, 2025 09:58:05 UTC          September 03, 2025 09:58:25 UTC          Success
Creating repositories using yum          September 03, 2025 09:58:25 UTC          September 03, 2025 09:58:27 UTC          Success
Updating YumPluginVersionLock rpm        September 03, 2025 09:58:27 UTC          September 03, 2025 09:58:29 UTC          Success
Applying OS Patches                      September 03, 2025 09:58:29 UTC          September 03, 2025 09:59:03 UTC          Success
Creating repositories using yum          September 03, 2025 09:59:03 UTC          September 03, 2025 09:59:03 UTC          Success
Applying HMP Patches                     September 03, 2025 09:59:04 UTC          September 03, 2025 09:59:06 UTC          Success
Applying Firmware local Disk Patches     September 03, 2025 09:59:07 UTC          September 03, 2025 09:59:10 UTC          Success
Oda-hw-mgmt upgrade                      September 03, 2025 09:59:07 UTC          September 03, 2025 09:59:07 UTC          Success
Patch location validation                September 03, 2025 09:59:07 UTC          September 03, 2025 09:59:07 UTC          Success
Applying Firmware local Controller Patch September 03, 2025 09:59:10 UTC          September 03, 2025 09:59:15 UTC          Success
Checking Ilom patch Version              September 03, 2025 09:59:15 UTC          September 03, 2025 09:59:15 UTC          Success
Patch location validation                September 03, 2025 09:59:15 UTC          September 03, 2025 09:59:15 UTC          Success
Save password in Wallet                  September 03, 2025 09:59:15 UTC          September 03, 2025 09:59:16 UTC          Success
Apply Ilom patch                         September 03, 2025 09:59:16 UTC          September 03, 2025 10:09:24 UTC          Success
Disabling IPMI v2                        September 03, 2025 09:59:16 UTC          September 03, 2025 09:59:16 UTC          Success
Cleanup JRE Home                         September 03, 2025 10:09:24 UTC          September 03, 2025 10:09:24 UTC          Success
Copying Flash Bios to Temp location      September 03, 2025 10:09:24 UTC          September 03, 2025 10:09:24 UTC          Success
Generating and saving BOM                September 03, 2025 10:09:24 UTC          September 03, 2025 10:09:27 UTC          Success
Update System version                    September 03, 2025 10:09:24 UTC          September 03, 2025 10:09:24 UTC          Success
PreRebootNode Actions                    September 03, 2025 10:09:27 UTC          September 03, 2025 10:09:27 UTC          Success
Reboot Node                              September 03, 2025 10:09:27 UTC          September 03, 2025 10:09:27 UTC          Success

The system will reboot once patched, let’s check the versions:

odacli describe-component
System Version
--------------
19.27.0.0.0

System Node Name
----------------
oak

Local System Version
--------------------
19.27.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                      19.27.0.0.0          up-to-date

DCSCONTROLLER                            19.27.0.0.0          up-to-date

DCSCLI                                   19.27.0.0.0          up-to-date

DCSAGENT                                 19.27.0.0.0          up-to-date

DCSADMIN                                 19.27.0.0.0          up-to-date

OS                                       8.10                 up-to-date

ILOM                                     5.1.4.25.r160118     up-to-date

BIOS                                     52140100             up-to-date

LOCAL CONTROLLER FIRMWARE {
     [c8, c9]                            214.2.271.9          up-to-date
}

SHARED CONTROLLER FIRMWARE {
     [c0, c1, c2, c3, c4, c5]            VDV1RL06             up-to-date
}

LOCAL DISK FIRMWARE {
     [c6d0, c6d1]                        XC311102             up-to-date
}

HMP                                      2.4.10.1.600         up-to-date

No need to apply the storage patch here, but if needed, apply it with:

odacli update-storage -v 19.27.0.0.0

Provision the appliance

Now it’s time to put back Oracle binaries on this system. First register the GI and DB clones, then do the create-appliance with the json file you prepared earlier:

cd /opt/dbi
unzip p30403662_1927000_Linux-x86-64.zip 
unzip p30403673_1927000_Linux-x86-64.zip

odacli update-repository -f /opt/dbi/odacli-dcs-19.27.0.0.0-250527-GI-19.27.0.0.zip
sleep 90 ; odacli describe-job -i c8f3559f-7f93-4df1-ba99-7511111f27ce

Job details
----------------------------------------------------------------
                     ID:  c8f3559f-7f93-4df1-ba99-7511111f27ce
            Description:  Repository Update
                 Status:  Success
                Created:  September 03, 2025 10:19:04 UTC
                Message:  /opt/dbi/odacli-dcs-19.27.0.0.0-250527-GI-19.27.0.0.zip

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- -------
Unzip bundle                             September 03, 2025 10:19:04 UTC          September 03, 2025 10:20:08 UTC          Success

odacli update-repository -f /opt/dbi/odacli-dcs-19.27.0.0.0-250527-DB-19.27.0.0.zip
sleep 90 ; odacli describe-job -i 96187596-16f6-413c-8905-9d85c6d3e106

Job details
----------------------------------------------------------------
                     ID:  96187596-16f6-413c-8905-9d85c6d3e106
            Description:  Repository Update
                 Status:  Success
                Created:  September 03, 2025 10:20:50 UTC
                Message:  /opt/dbi/odacli-dcs-19.27.0.0.0-250527-DB-19.27.0.0.zip

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- -------
Unzip bundle                             September 03, 2025 10:20:50 UTC          September 03, 2025 10:21:46 UTC          Success
 

odacli create-appliance -r /opt/dbi/deploy_odaprdcr.txt

Enter an initial password for Web Console account (oda-admin): **********
Confirm the password for Web Console account (oda-admin): **********
User 'oda-admin' created successfully...

odacli describe-job -i 22fec8d6-d9ae-4576-8f18-d8e1a0b00370
Job details
----------------------------------------------------------------
                     ID:  22fec8d6-d9ae-4576-8f18-d8e1a0b00370
            Description:  Provisioning service creation
                 Status:  Success
                Created:  September 03, 2025 12:45:37 CEST
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- -------
Validate absence of Interconnect         September 03, 2025 12:45:40 CEST         September 03, 2025 12:45:40 CEST         Success
network configuration file
Restart network interface pubnet         September 03, 2025 12:45:43 CEST         September 03, 2025 12:45:49 CEST         Success
Setting up Network                       September 03, 2025 12:45:43 CEST         September 03, 2025 12:45:43 CEST         Success
Setting up Vlan                          September 03, 2025 12:45:50 CEST         September 03, 2025 12:45:51 CEST         Success
Restart network interface priv0.100      September 03, 2025 12:45:51 CEST         September 03, 2025 12:45:52 CEST         Success
Restart network interface privasm        September 03, 2025 12:45:52 CEST         September 03, 2025 12:45:53 CEST         Success
Restart network interface privasm        September 03, 2025 12:45:53 CEST         September 03, 2025 12:45:59 CEST         Success
Setting up Network                       September 03, 2025 12:45:53 CEST         September 03, 2025 12:45:53 CEST         Success
Network update                           September 03, 2025 12:45:59 CEST         September 03, 2025 12:46:13 CEST         Success
Restart network interface btbond1        September 03, 2025 12:45:59 CEST         September 03, 2025 12:46:03 CEST         Success
Setting up Network                       September 03, 2025 12:45:59 CEST         September 03, 2025 12:45:59 CEST         Success
Updating network                         September 03, 2025 12:45:59 CEST         September 03, 2025 12:46:13 CEST         Success
Restart network interface btbond1        September 03, 2025 12:46:03 CEST         September 03, 2025 12:46:07 CEST         Success
Restart network interface pubnet         September 03, 2025 12:46:07 CEST         September 03, 2025 12:46:13 CEST         Success
OS usergroup 'asmadmin' creation         September 03, 2025 12:46:13 CEST         September 03, 2025 12:46:13 CEST         Success
OS usergroup 'asmdba' creation           September 03, 2025 12:46:13 CEST         September 03, 2025 12:46:13 CEST         Success
OS usergroup 'asmoper' creation          September 03, 2025 12:46:13 CEST         September 03, 2025 12:46:13 CEST         Success
OS usergroup 'dba' creation              September 03, 2025 12:46:13 CEST         September 03, 2025 12:46:14 CEST         Success
Validate availability of pubnet          September 03, 2025 12:46:13 CEST         September 03, 2025 12:46:13 CEST         Success
OS user 'grid' creation                  September 03, 2025 12:46:14 CEST         September 03, 2025 12:46:15 CEST         Success
OS usergroup 'dbaoper' creation          September 03, 2025 12:46:14 CEST         September 03, 2025 12:46:14 CEST         Success
OS usergroup 'oinstall' creation         September 03, 2025 12:46:14 CEST         September 03, 2025 12:46:14 CEST         Success
OS user 'oracle' creation                September 03, 2025 12:46:15 CEST         September 03, 2025 12:46:16 CEST         Success
Add SYSNAME in Env                       September 03, 2025 12:46:16 CEST         September 03, 2025 12:46:16 CEST         Success
Backup Config name validation            September 03, 2025 12:46:16 CEST         September 03, 2025 12:46:16 CEST         Success
Backup config metadata persist           September 03, 2025 12:46:16 CEST         September 03, 2025 12:46:16 CEST         Success
Default backup policy creation           September 03, 2025 12:46:16 CEST         September 03, 2025 12:46:16 CEST         Success
Grant permission to RHP files            September 03, 2025 12:46:16 CEST         September 03, 2025 12:46:16 CEST         Success
Install oracle-ahf                       September 03, 2025 12:46:16 CEST         September 03, 2025 12:47:23 CEST         Success
Stop DCS Admin                           September 03, 2025 12:48:25 CEST         September 03, 2025 12:48:26 CEST         Success
Generate mTLS certificates               September 03, 2025 12:48:26 CEST         September 03, 2025 12:48:27 CEST         Success
 
Exporting Public Keys                    September 03, 2025 12:48:27 CEST         September 03, 2025 12:48:28 CEST         Success
Creating Trust Store                     September 03, 2025 12:48:29 CEST         September 03, 2025 12:48:31 CEST         Success
Restart DCS Admin                        September 03, 2025 12:48:31 CEST         September 03, 2025 12:48:52 CEST         Success
Update config files                      September 03, 2025 12:48:31 CEST         September 03, 2025 12:48:31 CEST         Success
Creating GI home directories             September 03, 2025 12:48:52 CEST         September 03, 2025 12:48:52 CEST         Success
Extract GI clone                         September 03, 2025 12:48:52 CEST         September 03, 2025 12:50:11 CEST         Success
Grid home creation                       September 03, 2025 12:48:52 CEST         September 03, 2025 12:50:13 CEST         Success
Storage discovery                        September 03, 2025 12:50:13 CEST         September 03, 2025 12:51:33 CEST         Success
Creating wallet for Root User            September 03, 2025 12:51:33 CEST         September 03, 2025 12:51:36 CEST         Success
Creating wallet for ASM Client           September 03, 2025 12:51:36 CEST         September 03, 2025 12:51:39 CEST         Success
Grid stack creation                      September 03, 2025 12:51:39 CEST         September 03, 2025 13:00:38 CEST         Success
Provisioning GI with RHP                 September 03, 2025 12:51:39 CEST         September 03, 2025 12:57:27 CEST         Success
Set CRS ping target                      September 03, 2025 12:57:28 CEST         September 03, 2025 12:57:29 CEST         Success
Updating GI home version                 September 03, 2025 12:57:29 CEST         September 03, 2025 12:57:34 CEST         Success
Restarting Clusterware                   September 03, 2025 12:57:35 CEST         September 03, 2025 13:00:38 CEST         Success
Post cluster OAKD configuration          September 03, 2025 13:00:38 CEST         September 03, 2025 13:01:33 CEST         Success
Disk group 'RECO' creation               September 03, 2025 13:01:41 CEST         September 03, 2025 13:01:52 CEST         Success
Setting ACL for disk groups              September 03, 2025 13:01:52 CEST         September 03, 2025 13:01:55 CEST         Success
Modify DB file attributes                September 03, 2025 13:01:55 CEST         September 03, 2025 13:02:03 CEST         Success
Register Scan and Vips to Public Network September 03, 2025 13:02:03 CEST         September 03, 2025 13:02:05 CEST         Success
Configure export clones resource         September 03, 2025 13:03:24 CEST         September 03, 2025 13:03:24 CEST         Success
Volume 'commonstore' creation            September 03, 2025 13:03:24 CEST         September 03, 2025 13:03:41 CEST         Success
ACFS File system 'DATA' creation         September 03, 2025 13:03:41 CEST         September 03, 2025 13:04:03 CEST         Success
Creating ACFS database home              September 03, 2025 13:04:04 CEST         September 03, 2025 13:05:22 CEST         Success
Database home creation                   September 03, 2025 13:04:04 CEST         September 03, 2025 13:07:55 CEST         Success
Validating dbHome available space        September 03, 2025 13:05:23 CEST         September 03, 2025 13:05:23 CEST         Success
Create required directories              September 03, 2025 13:05:24 CEST         September 03, 2025 13:05:24 CEST         Success
Creating DbHome Directory                September 03, 2025 13:05:24 CEST         September 03, 2025 13:05:24 CEST         Success
Extract DB clone                         September 03, 2025 13:05:24 CEST         September 03, 2025 13:06:18 CEST         Success
ProvDbHome by using RHP                  September 03, 2025 13:06:18 CEST         September 03, 2025 13:07:33 CEST         Success
Enable DB options                        September 03, 2025 13:07:33 CEST         September 03, 2025 13:07:47 CEST         Success
Creating wallet for DB Client            September 03, 2025 13:07:51 CEST         September 03, 2025 13:07:55 CEST         Success
Database Creation by RHP                 September 03, 2025 13:07:56 CEST         September 03, 2025 13:20:28 CEST         Success
Provisioning service creation            September 03, 2025 13:07:56 CEST         September 03, 2025 13:23:47 CEST         Success
Change permission for xdb wallet files   September 03, 2025 13:20:28 CEST         September 03, 2025 13:20:29 CEST         Success
Add Startup Trigger to Open all PDBS     September 03, 2025 13:20:29 CEST         September 03, 2025 13:20:30 CEST         Success
Place SnapshotCtrlFile in sharedLoc      September 03, 2025 13:20:30 CEST         September 03, 2025 13:20:33 CEST         Success
SqlPatch upgrade                         September 03, 2025 13:22:08 CEST         September 03, 2025 13:22:30 CEST         Success
Running dbms_stats init_package          September 03, 2025 13:22:30 CEST         September 03, 2025 13:22:31 CEST         Success
Set log_archive_dest for Database        September 03, 2025 13:22:31 CEST         September 03, 2025 13:22:34 CEST         Success
Updating the Database version            September 03, 2025 13:22:34 CEST         September 03, 2025 13:22:38 CEST         Success
Users tablespace creation                September 03, 2025 13:23:47 CEST         September 03, 2025 13:23:50 CEST         Success
Clear all listeners from Database        September 03, 2025 13:23:50 CEST         September 03, 2025 13:23:50 CEST         Success
Clear local and remote listeners from    September 03, 2025 13:23:52 CEST         September 03, 2025 13:23:53 CEST         Success
Database
Copy Pwfile to Shared Storage            September 03, 2025 13:23:53 CEST         September 03, 2025 13:23:59 CEST         Success
Persist new agent state entry            September 03, 2025 13:24:02 CEST         September 03, 2025 13:24:02 CEST         Success
Persist new agent state entry            September 03, 2025 13:24:02 CEST         September 03, 2025 13:24:02 CEST         Success
Provisioning service creation            September 03, 2025 13:24:02 CEST         September 03, 2025 13:24:02 CEST         Success
Restart DCS Agent                        September 03, 2025 13:24:02 CEST         September 03, 2025 13:24:03 CEST         Success

Creating the appliance usually takes 40 to 50 minutes.

Check core reduction

Reimaging an ODA will keep the core reduction, but make sure it’s true with these commands:

odacli list-cpucores
Node  Cores  Modified                           Job Status
----- ------ ---------------------------------- ---------------
0     4      September 3, 2025 9:46:11 AM CEST  CONFIGURED

lscpu | grep Core
Core(s) per socket:  2

Here I have 4 enabled cores, meaning 2 processor licenses.

Setup older DB home

As I will bring back my Data Guard setup with 19.20 databases, I will need a 19.20 DB home. Let’s register the clone file and create the DB home. An increase of the diagnostic dest is probably needed, 10GB is way too small for me:

odacli list-dbhome-storages
=============================================================================================================
ID                                     Node Description          Disk Group Volume      Size(GB)   Status
-------------------------------------- ---- -------------------- ---------- ----------- ---------- ----------
bef3ae8a-96d3-44d4-b179-1e5dbdd2a939   0    ORACLE_HOME          DATA       orahome_sh  80         CONFIGURED
e972cc17-ff92-4203-b23f-55fab02a7cc0   0    ORACLE_BASE          DATA       odabase_n0  10         CONFIGURED
=============================================================================================================

odacli modify-dbhome-storage -i e972cc17-ff92-4203-b23f-55fab02a7cc0 -s 50

sleep 10 ; odacli list-dbhome-storages
=============================================================================================================
ID                                     Node Description          Disk Group Volume      Size(GB)   Status
-------------------------------------- ---- -------------------- ---------- ----------- ---------- ----------
bef3ae8a-96d3-44d4-b179-1e5dbdd2a939   0    ORACLE_HOME          DATA       orahome_sh  80         CONFIGURED
e972cc17-ff92-4203-b23f-55fab02a7cc0   0    ORACLE_BASE          DATA       odabase_n0  50         CONFIGURED
=============================================================================================================


cd /opt/dbi
unzip p30403662_1920000_Linux-x86-64.zip 
odacli update-repository -f /opt/dbi/odacli-dcs-19.20.0.0.0-230720-DB-19.20.0.0.zip

sleep 30 ; odacli describe-job -i 3ee5db4d-d350-47b4-8e24-18949d244ab7
Job details
----------------------------------------------------------------
                     ID:  3ee5db4d-d350-47b4-8e24-18949d244ab7
            Description:  Repository Update
                 Status:  Success
                Created:  September 03, 2025 13:47:23 CEST
                Message:  /opt/dbi/odacli-dcs-19.20.0.0.0-230720-DB-19.20.0.0.zip

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- -------
Unzip bundle                             September 03, 2025 13:47:23 CEST         September 03, 2025 13:47:55 CEST         Success

odacli create-dbhome -v 19.20.0.0.230718

sleep 240 ; odacli describe-job -i 3123544b-60c5-418a-82b5-c110dd68aafa
Job details
----------------------------------------------------------------
                     ID:  3123544b-60c5-418a-82b5-c110dd68aafa
            Description:  Database Home OraDB19000_home2 creation with version :19.20.0.0.230718
                 Status:  Success
                Created:  September 03, 2025 13:53:21 CEST
                Message:  Create Database Home

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- -------
Setting up SSH equivalence               September 03, 2025 13:53:32 CEST         September 03, 2025 13:53:34 CEST         Success
Setting up SSH equivalence               September 03, 2025 13:53:34 CEST         September 03, 2025 13:53:35 CEST         Success
Creating ACFS database home              September 03, 2025 13:53:35 CEST         September 03, 2025 13:53:35 CEST         Success
Validating dbHome available space        September 03, 2025 13:53:35 CEST         September 03, 2025 13:53:35 CEST         Success
Create required directories              September 03, 2025 13:53:36 CEST         September 03, 2025 13:53:36 CEST         Success
Creating DbHome Directory                September 03, 2025 13:53:36 CEST         September 03, 2025 13:53:36 CEST         Success
Extract DB clone                         September 03, 2025 13:53:36 CEST         September 03, 2025 13:54:23 CEST         Success
ProvDbHome by using RHP                  September 03, 2025 13:54:23 CEST         September 03, 2025 13:55:39 CEST         Success
Enable DB options                        September 03, 2025 13:55:39 CEST         September 03, 2025 13:55:54 CEST         Success
Creating wallet for DB Client            September 03, 2025 13:55:57 CEST         September 03, 2025 13:55:58 CEST         Success

odacli list-dbhomes
ID                                       Name                 DB Version           DB Edition Home Location                                            Status
---------------------------------------- -------------------- -------------------- ---------- -------------------------------------------------------- ----------
19ee71be-07ae-4373-9bb6-31db8195e36c     OraDB19000_home1     19.27.0.0.250415     EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1     CONFIGURED
cd2dc372-3f95-40f1-917c-ee0793c26664     OraDB19000_home2     19.20.0.0.230718     EE         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2     CONFIGURED

Create the databases

Your ODA registry needs to know the databases, so I will create them before doing the restore. I don’t use the -io (instance only) of odacli for database creation as srvctl entries will be missing. So let’s create a new standalone database and then remove its files:

odacli create-database -cs AL32UTF8 -no-c -u BMC3PRD_CR -dh 'cd2dc372-3f95-40f1-917c-ee0793c26664' -n BMC3PRD -s odb1s -l AMERICAN -dt AMERICA -no-co -r asm

sleep 660 ; odacli describe-job -i 40b80359-0607-4330-ae6e-055d284a780e
Job details
----------------------------------------------------------------
                     ID:  40b80359-0607-4330-ae6e-055d284a780e
            Description:  Database service creation with DB name: BMC3PRD
                 Status:  Success
                Created:  September 03, 2025 15:11:48 CEST
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- -------
Database Creation by RHP                 September 03, 2025 15:11:50 CEST         September 03, 2025 15:19:24 CEST         Success
Database Service creation                September 03, 2025 15:11:50 CEST         September 03, 2025 15:22:19 CEST         Success
Setting up SSH equivalence               September 03, 2025 15:11:50 CEST         September 03, 2025 15:11:50 CEST         Success
Setting up SSH equivalence               September 03, 2025 15:11:50 CEST         September 03, 2025 15:11:50 CEST         Success
Change permission for xdb wallet files   September 03, 2025 15:19:24 CEST         September 03, 2025 15:19:25 CEST         Success
Place SnapshotCtrlFile in sharedLoc      September 03, 2025 15:19:25 CEST         September 03, 2025 15:19:28 CEST         Success
SqlPatch upgrade                         September 03, 2025 15:20:43 CEST         September 03, 2025 15:20:50 CEST         Success
Running dbms_stats init_package          September 03, 2025 15:20:50 CEST         September 03, 2025 15:20:51 CEST         Success
Set log_archive_dest for Database        September 03, 2025 15:20:51 CEST         September 03, 2025 15:20:54 CEST         Success
Updating the Database version            September 03, 2025 15:20:54 CEST         September 03, 2025 15:20:58 CEST         Success
Create Users tablespace                  September 03, 2025 15:22:19 CEST         September 03, 2025 15:22:20 CEST         Success
Clear all listeners from Database        September 03, 2025 15:22:20 CEST         September 03, 2025 15:22:21 CEST         Success
Copy Pwfile to Shared Storage            September 03, 2025 15:22:23 CEST         September 03, 2025 15:22:27 CEST         Success

su - oracle
. oraenv <<< BMC3PRD
sqlplus / as sysdba
set pages 0
set feedback off
spool /tmp/rmfiles.sh
select 'asmcmd rm -fr '||file_name from dba_data_files;
select 'asmcmd rm -fr '||file_name from dba_temp_files;
select 'asmcmd rm -fr '||member from v$logfile;
select 'asmcmd rm -fr '||name from v$controlfile;
select 'asmcmd rm -fr '||value from v$parameter where name='spfile';
spool off
shutdown abort;
exit
exit
su - grid
sh /tmp/rmfiles.sh
exit

Do the same for the other databases. Don’t try to create your databases with multiple odacli jobs running at the same time: this is something ODA cannot do in parallel, creation will be serialized and jobs will be waiting for the current one to complete.

Restore the databases

I’m used to move the spfile from ASM to the filesystem, so it’s the exact same path as the other server. In the same time, I change the role of the database as odacli created it as a primary:

su - oracle
. oraenv <<< BMC3PRD
srvctl modify database -db BMC3PRD_CR -spfile /u01/app/oracle/admin/BMC3PRD/pfile/spfileBMC3PRD.ora -startoption mount -role PHYSICAL_STANDBY

Now let’s restore the spfile and the controlfile. RMAN is able to start an instance without any spfile for recovering the spfile, it’s very convenient:

rman target / 

startup nomount;
restore spfile from '/backup/Patch19.27/ctl_spf/spf_BMC3PRD_20250903.bck';
Starting restore at 04-SEP-25
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=433 device type=DISK
channel ORA_DISK_1: restoring spfile from AUTOBACKUP /backup/Patch19.27/ctl_spf/spf_BMC3PRD_20250903.bck
channel ORA_DISK_1: SPFILE restore from AUTOBACKUP complete
Finished restore at 04-SEP-25
 
startup force nomount;

restore controlfile from '/backup/Patch19.27/ctl_spf/ctl_BMC3PRD_20250903.bck';

Starting restore at 04-SEP-25
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=26 device type=DISK
channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:05
output file name=+RECO/BMC3PRD_CR/CONTROLFILE/current.486.1210937913
Finished restore at 04-SEP-25

Now I can start restoring the database. Choose the parallel degree carefully: as your ODA has nothing else to do, I would recommend using the maximum channels you could for big databases, for example between 4 and 8 channels for an ODA with 4 enabled cores:

startup force mount;
run {
allocate channel c1 device type disk;
allocate channel c2 device type disk;
allocate channel c3 device type disk;
allocate channel c4 device type disk;
restore database ;
}


…
channel c4: restoring datafile 00004 to +DATA/BMC3PRD/DATAFILE/undotbs1.291.1042201603
channel c4: restoring datafile 00005 to +DATA/BMC3PRD/DATAFILE/users.294.1042202001
channel c4: reading from backup piece /backup/BMC3PRD/backupset/20250901_000002_inc0_BMC3PRD_45931446_s36914_p1.bck
channel c3: piece handle=/backup/BMC3PRD/backupset/20250901_000002_inc0_BMC3PRD_45931446_s36908_p2.bck tag=INC0_20250901_000002
channel c3: restored backup piece 2
channel c3: restore complete, elapsed time: 00:00:38
channel c1: piece handle=/backup/BMC3PRD/backupset/20250901_000002_inc0_BMC3PRD_45931446_s36911_p2.bck tag=INC0_20250901_000002
channel c1: restored backup piece 2
channel c1: restore complete, elapsed time: 00:00:42
channel c4: piece handle=/backup/BMC3PRD/backupset/20250901_000002_inc0_BMC3PRD_45931446_s36914_p1.bck tag=INC0_20250901_000002
channel c4: restored backup piece 1
channel c4: restore complete, elapsed time: 00:00:35
channel c2: piece handle=/backup/BMC3PRD/backupset/20250901_000002_inc0_BMC3PRD_45931446_s36911_p1.bck tag=INC0_20250901_000002
channel c2: restored backup piece 1
channel c2: restore complete, elapsed time: 00:00:54
Finished restore at 04-SEP-25
released channel: c1
released channel: c2
released channel: c3
released channel: c4

exit

Add TNS entries for Data Guard and a static entry for the listener

Your database is now restored, you’ll need to configure communication with the primary. Let’s add the following TNS entries:

vi $ORACLE_HOME/network/admin/tnsnames.ora
…

BMC3PRD_CR =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = odaprdcr)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = BMC3PRD_CR)
    )
  )

BMC3PRD_IN =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = odaprdin)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = BMC3PRD_IN)
    )
  )

You will also need the static entry for your database in the listener.ora file:

su - grid
vi $ORACLE_HOME/network/admin/listener.ora
SID_LIST_LISTENER=
(SID_LIST=
  (SID_DESC=
    (GLOBAL_DBNAME = BMC3PRD_CR)
    (ORACLE_HOME = /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2)
    (SID_NAME = BMC3PRD)
  )
)
exit

Copy the password file and modify its location

Pick up the password file from primary server and relocate the file to the filesystem with srvctl:

scp oracle@odaprdin:/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2/dbs/orapwBMC3PRD /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2/dbs/

srvctl modify database -db BMC3PRD_CR -pwfile
srvctl stop database -db BMC3PRD_CR ; sleep 10 ; srvctl start database -db BMC3PRD_CR -o mount

You can use a symbolic link in the dbs subfolder if you want to put this password file alongside your spfile.

Recover the database

Restored datafiles are probably several hours old, so let’s do a recover on the standby from the primary service before configuring Data Guard:

rman target sys
recover database from service 'BMC3PRD_IN';
…
channel ORA_DISK_1: using network backup set from service BMC3PRD_IN
destination for restore of datafile 00007: +DATA/BMC3PRD_CR/DATAFILE/dm_mc3prd_docbase.477.1210938063
channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: using network backup set from service BMC3PRD_IN
destination for restore of datafile 00008: +DATA/BMC3PRD_CR/DATAFILE/dm_mc3prd_docbase.478.1210938063
channel ORA_DISK_1: restore complete, elapsed time: 00:00:08
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: using network backup set from service BMC3PRD_IN
destination for restore of datafile 00009: +DATA/BMC3PRD_CR/DATAFILE/dm_mc3prd_docbase.479.1210938067
channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: using network backup set from service BMC3PRD_IN
destination for restore of datafile 00010: +DATA/BMC3PRD_CR/DATAFILE/dm_mc3prd_docbase.480.1210938165
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07

starting media recovery

media recovery complete, elapsed time: 00:00:00
Finished recover at 04-SEP-25

exit

If archivelog files are missing or if recovery is not possible, it doesn’t mean that it’s a problem. Data Guard should be able to solve a small or bigger archive gap.

Rebuild Data Guard configuration

Instead of adding back my standby database to the existing Data Guard configuration, I will recreate this configuration. It’s a matter of seconds:

dgmgrl sys
*****
Remove configuration;
create configuration DTG as primary database is 'BMC3PRD_IN' connect identifier is 'BMC3PRD_IN';
add database 'BMC3PRD_CR' as connect identifier is 'BMC3PRD_CR';
enable configuration;
edit database 'BMC3PRD_CR' set property LogXptMode='SYNC';
edit database 'BMC3PRD_IN' set property LogXptMode='SYNC';
edit database 'BMC3PRD_CR' set property StandbyFileManagement='AUTO';
edit database 'BMC3PRD_IN' set property StandbyFileManagement='AUTO';
EDIT DATABASE 'BMC3PRD_CR' SET PROPERTY 'ArchiveLagTarget'=1200; 
EDIT DATABASE 'BMC3PRD_IN' SET PROPERTY 'ArchiveLagTarget'=1200; 
edit configuration set protection mode as maxavailability;

Depending on the lag between your primary and this standby, it can takes several minutes to bring back the sync of both databases:

show configuration lag;
Configuration - dtg

  Protection Mode: MaxAvailability
  Members:
  BMC3PRD_IN - Primary database
    BMC3PRD_CR - Physical standby database
                  Transport Lag:      0 seconds (computed 1 second ago)
                  Apply Lag:          0 seconds (computed 1 second ago)

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 6 seconds ago)

This database is OK, let’s continue with the next ones.

Additional settings

If you made additional settings on your ODA, you’ll have to configure them again. It could be:

monitoring
additional network interfaces
additional services
mount points and shares
firewall rules
kerberos authentication
compatibility modes in sqlnet.ora file
…

Conclusion

With several multi-TB databases to restore, it took me less than 3 days per ODA for this operation. And these ODAs are now clean and ready for their second life.

Reimaging an ODA from scratch is a good solution as soon as a Data Guard setup allows the databases to run on another ODA for several days. And with a good preparation, the job is not that huge. And there is much less troubleshooting compared to patching. This is something to consider.

L’article Reimage your ODA from scratch est apparu en premier sur dbi Blog.

PRGH-1030 when doing restore-node -g on ODA

Jérôme Dubar — Fri, 14 Mar 2025 12:47:55 +0000

Introduction

Patching your Oracle Database Appliance from 19.20 or earlier release to 19.21 or newer release implies the use of Data Preserving Reprovisioning (DPR). Most of the time, and with an adequate preparation, the DPR works fine. But if something goes wrong at the restore-node -g step, you will need some troubleshooting and maybe opening a SR. There is no possible rollback. Here is the problem I recently had when patching an ODA from 19.20 to 19.26 using the intermediate 19.24 for DPR.

What is DPR and why you need it on ODA?

Oracle Database Appliance has a global quarterly patch: this is the only patch you are supposed to use on this kind of hardware. The patch includes updates for Grid Infrastructure (GI) and database (DB), but also updates for the operating system (OS), BIOS, firmwares and other software components. Regarding the OS, Linux 7 was in use up to 19.20 version. Starting from 19.21, Linux 8 is mandatory and there is no update-server command available: you will need to reinstall the new OS on your ODA using DPR.

Basically, DPR is a complete reinstall of the OS without erasing the data disks. Before doing the reinstall, you will “detach” the node. After the reinstall, you will “restore” the node using an odacli command: restore-node -g. You can compare this to the unplug/plug operations on a PDB. Unplugging a PDB is writing the metadata in a file you will later use to plug in back your PDB somewhere else. The DPR feature is embedded in patch 19.21, 19.22, 19.23 and 19.24. As patches are only cumulative with the 4 previous versions, starting from 19.25 you’re supposed to use release 19.21 at least, meaning already in Linux 8.

My ODA setup: a real client environment

My client has 2x ODA X9-2S with Enterprise Edition and Data Guard for production databases. These ODAs were deployed 2 years ago, and already patched from 19.17 to 19.20 1 year ago. Now, it’s time to patch them to the latest 19.26, using DPR 19.24 for the Linux 8 jump.

Pre-upgrade report and detach-node

DCS components are updated to 19.24 at first step, then the pre-upgrade report can be tried:

odacli update-dcsadmin -v 19.24.0.0.0
odacli update-dcscomponents -v 19.24.0.0.0
odacli update-dcsagent -v 19.24.0.0.0
odacli create-preupgradereport -bm 
odacli describe-preupgradereport -i 03f53c9c-fe82-4c2b-bf18-49fd31853054
...

This pre-upgrade report MUST be successful: if not, solve the listed problems and retry until it’s OK.

Once OK, the detach-node will backup the metadata of your ODA for later restore:

odacli detach-node -all

As detach files are stored locally, make sure to backup these files on an external volume otherwise you will not be able to restore your data after reimaging.

cp -r /opt/oracle/oak/restore/out/* /mnt/backup/ODA_backup/

Reimaging the ODA

Reimaging is done with patch 30403643 (version 19.24 in my case). This is an ISO file you will virtually plug in through ILOM as a virtual CDROM. Then you will reboot the server and automatic setup will start. After this reimaging, you will need to do a configure-firstnet for basic network configuration, and then register the new GI and the detached server archive:

odacli update-repository -f /mnt/backup/ODA_backup/odacli-dcs-19.24.0.0.0-240724-GI-19.24.0.0.zip
odacli update-repository -f /mnt/backup/ODA_backup/serverarchive_srvoda1.zip

Once both files have successfully been registered, the restore-node can be done:

odacli restore-node -g

PRGH-1030 and timeout of the restore-node -g

The restore-node -g is similar to a create-appliance: it will configure the system, create the users, provision GI and configure ASM. Instead of configuring ASM with fresh disks without any data, it will read the ASM headers on disks and mount the existing diskgroups. It means that you will get back the DB homes, the databases and ACFS volumes.

This time, the restore-node -g didn’t work for me, it took a very long time (more than one hour) before ending with a failure, a timeout was probably triggered as odacli jobs always finish with a success or a failure:

odacli describe-job -i "46addc5f-7a1f-4f4b-bccf-78cb3708bef9"
Job details
----------------------------------------------------------------
                     ID:  46addc5f-7a1f-4f4b-bccf-78cb3708bef9
            Description:  Restore node service - GI
                 Status:  Failure (To view Error Correlation report, run "odacli describe-job -i 46addc5f-7a1f-4f4b-bccf-78cb3708bef9 --ecr" command)
                Created:  March 5, 2025 9:03:44 AM CET
                Message:  DCS-10001:Internal error encountered: Failed to provision GI with RHP at the home: /u01/app/19.24.0.0/grid: DCS-10001:Internal error encountered: PRGH-1030 : The environments on nodes 'srvoda1' do not satisfy some of the prerequisite checks.
..
 
Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ---------
Restore node service creation            March 5, 2025 9:04:01 AM CET             March 5, 2025 10:17:35 AM CET            Failure
Restore node service creation            March 5, 2025 9:04:01 AM CET             March 5, 2025 10:17:35 AM CET            Failure
Restore node service creation            March 5, 2025 9:04:01 AM CET             March 5, 2025 10:17:35 AM CET            Failure
Provisioning service creation            March 5, 2025 9:04:01 AM CET             March 5, 2025 10:17:35 AM CET            Failure
Provisioning service creation            March 5, 2025 9:04:01 AM CET             March 5, 2025 10:17:35 AM CET            Failure
Validate absence of Interconnect         March 5, 2025 9:04:01 AM CET             March 5, 2025 9:04:02 AM CET             Success
network configuration file
Setting up Network                       March 5, 2025 9:04:03 AM CET             March 5, 2025 9:04:03 AM CET             Success
Restart network interface pubnet         March 5, 2025 9:04:03 AM CET             March 5, 2025 9:04:09 AM CET             Success
Setting up Vlan                          March 5, 2025 9:04:09 AM CET             March 5, 2025 9:04:10 AM CET             Success
Restart network interface priv0.100      March 5, 2025 9:04:10 AM CET             March 5, 2025 9:04:11 AM CET             Success
Restart network interface privasm        March 5, 2025 9:04:11 AM CET             March 5, 2025 9:04:11 AM CET             Success
Setting up Network                       March 5, 2025 9:04:11 AM CET             March 5, 2025 9:04:11 AM CET             Success
Restart network interface privasm        March 5, 2025 9:04:11 AM CET             March 5, 2025 9:04:17 AM CET             Success
Network update                           March 5, 2025 9:04:17 AM CET             March 5, 2025 9:04:30 AM CET             Success
Updating network                         March 5, 2025 9:04:17 AM CET             March 5, 2025 9:04:30 AM CET             Success
Setting up Network                       March 5, 2025 9:04:17 AM CET             March 5, 2025 9:04:17 AM CET             Success
Restart network interface btbond1        March 5, 2025 9:04:17 AM CET             March 5, 2025 9:04:20 AM CET             Success
Restart network interface btbond1        March 5, 2025 9:04:20 AM CET             March 5, 2025 9:04:24 AM CET             Success
Restart network interface pubnet         March 5, 2025 9:04:24 AM CET             March 5, 2025 9:04:29 AM CET             Success
Validate availability of pubnet          March 5, 2025 9:04:30 AM CET             March 5, 2025 9:04:30 AM CET             Success
OS usergroup 'asmdba' creation           March 5, 2025 9:04:30 AM CET             March 5, 2025 9:04:30 AM CET             Success
OS usergroup 'asmoper' creation          March 5, 2025 9:04:30 AM CET             March 5, 2025 9:04:30 AM CET             Success
OS usergroup 'asmadmin' creation         March 5, 2025 9:04:30 AM CET             March 5, 2025 9:04:30 AM CET             Success
OS usergroup 'dba' creation              March 5, 2025 9:04:30 AM CET             March 5, 2025 9:04:30 AM CET             Success
OS usergroup 'dbaoper' creation          March 5, 2025 9:04:30 AM CET             March 5, 2025 9:04:30 AM CET             Success
OS usergroup 'oinstall' creation         March 5, 2025 9:04:30 AM CET             March 5, 2025 9:04:31 AM CET             Success
OS user 'grid' creation                  March 5, 2025 9:04:31 AM CET             March 5, 2025 9:04:31 AM CET             Success
OS user 'oracle' creation                March 5, 2025 9:04:31 AM CET             March 5, 2025 9:04:31 AM CET             Success
Default backup policy creation           March 5, 2025 9:04:31 AM CET             March 5, 2025 9:04:31 AM CET             Success
Backup Config name validation            March 5, 2025 9:04:31 AM CET             March 5, 2025 9:04:31 AM CET             Success
Backup config metadata persist           March 5, 2025 9:04:31 AM CET             March 5, 2025 9:04:31 AM CET             Success
Grant permission to RHP files            March 5, 2025 9:04:31 AM CET             March 5, 2025 9:04:31 AM CET             Success
Add SYSNAME in Env                       March 5, 2025 9:04:32 AM CET             March 5, 2025 9:04:32 AM CET             Success
Install oracle-ahf                       March 5, 2025 9:04:32 AM CET             March 5, 2025 9:06:13 AM CET             Success
Stop DCS Admin                           March 5, 2025 9:06:16 AM CET             March 5, 2025 9:06:16 AM CET             Success
Generate mTLS certificates               March 5, 2025 9:06:16 AM CET             March 5, 2025 9:06:17 AM CET             Success
Exporting Public Keys                    March 5, 2025 9:06:17 AM CET             March 5, 2025 9:06:18 AM CET             Success
Creating Trust Store                     March 5, 2025 9:06:18 AM CET             March 5, 2025 9:06:22 AM CET             Success
Update config files                      March 5, 2025 9:06:22 AM CET             March 5, 2025 9:06:22 AM CET             Success
Restart DCS Admin                        March 5, 2025 9:06:22 AM CET             March 5, 2025 9:06:43 AM CET             Success
Unzipping storage configuration files    March 5, 2025 9:06:43 AM CET             March 5, 2025 9:06:43 AM CET             Success
Reloading multipath devices              March 5, 2025 9:06:43 AM CET             March 5, 2025 9:06:44 AM CET             Success
Restart oakd                             March 5, 2025 9:06:44 AM CET             March 5, 2025 9:06:55 AM CET             Success
Restart oakd                             March 5, 2025 9:08:05 AM CET             March 5, 2025 9:08:15 AM CET             Success
Restore Quorum Disks                     March 5, 2025 9:08:15 AM CET             March 5, 2025 9:08:16 AM CET             Success
Creating GI home directories             March 5, 2025 9:08:16 AM CET             March 5, 2025 9:08:16 AM CET             Success
Extract GI clone                         March 5, 2025 9:08:16 AM CET             March 5, 2025 9:09:29 AM CET             Success
Creating wallet for Root User            March 5, 2025 9:09:29 AM CET             March 5, 2025 9:09:33 AM CET             Success
Creating wallet for ASM Client           March 5, 2025 9:09:33 AM CET             March 5, 2025 9:09:39 AM CET             Success
Grid stack creation                      March 5, 2025 9:09:39 AM CET             March 5, 2025 10:17:35 AM CET            Failure
GI Restore with RHP                      March 5, 2025 9:09:39 AM CET             March 5, 2025 10:17:35 AM CET            Failure

According to the error message, my system doesn’t satisfy the prerequisites checks.

Let’s check if error correlation may help:

odacli describe-job -i 46addc5f-7a1f-4f4b-bccf-78cb3708bef9 --ecr
ODA Assistant - Error Correlation report
----------------------------------------
          Failed job ID:  46addc5f-7a1f-4f4b-bccf-78cb3708bef9
            Description:  Restore node service - GI
             Start Time:  2025-03-05 09:04:01
               End Time:  2025-03-05 10:17:35
         EC report path: /opt/oracle/dcs/da/da_repo/46addc5f-7a1f-4f4b-bccf-78cb3708bef9.json

Failed Task Messages
--------------------
[Restore node service - GI] -  DCS-10001:Internal error encountered: Failed to provision GI with RHP at the home: /u01/app/19.24.0.0/grid: DCS-10001:Internal error encountered: PRGH-1030 : The environments on nodes 'srvoda1' do not satisfy some of the prerequisite checks. ..

srvoda1 Log Messages
----------------------------
  DCS Agent
  ~~~~~~~~~
    Error Logs
    ==========
    [Install oracle-ahf] - Trying to add string [ERROR] : Unable to switch to home directory
    [Install oracle-ahf] - error is package tfa-oda is not installed
    [GI Restore with RHP] - Calling rhp provGI
    [GI Restore with RHP] - Task(id: TaskDcsJsonRpcExt_707, name: GI Restore with RHP) failed
    [GI Restore with RHP] - .. d.subtasks.isempty=true d.status=Failure
    [Grid stack creation] - ..
    [Grid stack creation] - .. d.subtasks.isempty=false d.status=Failure
    [Grid stack creation] - DCS-10001:Internal error encountered: PRGH-1030 : The environments on nodes 'srvoda1' do not satisfy some of the prerequisite checks.
      Error code - DCS-10001
      Cause: An internal error occurred.
      Action: Contact Oracle Support for assistance.
    [Grid stack creation] - .., output:
    Warning Logs
    ============
    [[ SEND-THREAD 77 ]] - [ [ SEND-THREAD 77 ] dcs0-priv:22001] Request failed: Operation: GET Host: dcs0-priv:22001 Path: /joblocks/46addc5f-7a1f-4f4b-bccf-78cb3708bef9 Data: null Status: 404
    [Provisioning service creation] - dm_multipath module is not loaded, attempting to load it...

  RHP
  ~~~
    Error Logs
    ==========
    [jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - [SiteFactoryImpl.fetchSite:136]  EntityNotExistsException PRGR-110 : Repository object "SRVODA1" of type "SITE" does not exist.
    [jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - [WorkingCopyOperationImpl.internalAddGI:18517]  Expected: Site does not exist, will have to be created :PRGR-119 : Site "srvoda1" does not exist.
    [jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - [RemoteFactoryImpl.getCRSHomeOfRemoteCluster:1377]  COException: PRCZ-4001 : failed to execute command "/bin/cat" using the privileged execution plugin "odaexec" on nodes "srvoda1" within 120 seconds
    [jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - PRCZ-2103 : Failed to execute command "/bin/cat" on node "srvoda1" as user "root". Detailed error:
    [jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - /bin/cat: /etc/oracle/olr.loc: No such file or directory
    [jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - [WorkingCopyOperationImpl.internalAddGI:18972]  Exec Exception, expected here :PRGR-118 : Working copy "OraGrid192400" does not exist.
    [jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - [OracleHomeImpl.addWCMarkerFile:4403]  Ignore the exception to create RHP WC marker file : PRKH-1009 : CRS HOME must be defined in the environment or in the Oracle Cluster Registry
    [jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - [ExecCommandNoUserEqImpl.runCmd:491]  Final CompositeOperation exception: PRCC-1021 : One or more of the submitted commands did not execute successfully.
    [jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - PRCC-1026 : Command "/u01/app/19.24.0.0/grid/gridSetup.sh -silent -responseFile /u01/app/19.24.0.0/grid//crs/install/rhpdata/grid7317919467403765800.rsp -ignorePrereq -J-Dskip.cvu.root.checks=true -J-Doracle.install.grid.validate.all=false oracle_install_crs_ODA_CONFIG=olite oracle_install_crs_ConfigureMgmtDB=false -J-Doracle.install.crs.skipGIMRDiskGroupSizeCheck=true oracle_install_asm_UseExistingDG=true -J-Doracle.install.grid.validate.CreateASMDiskGroup=false" submitted on node srvoda1 timed out after 4,000 seconds.
    [jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - 
    [jobid-46addc5f-7a1f-4f4b-bccf-78cb3708bef9] - [OperationAPIImpl.provGI:676]  OperationAPIException: PRGH-1030 : The environments on nodes 'srvoda1' do not satisfy some of the prerequisite checks.

Release Notes
-------------
  No matching results were found.

Documentation
-------------
  1. Error in restore node process in Data Preserving Reprovisioning
      Abstract - In the Data Preserving Reprovisioning process, during node restore, an error may be encountered.
      Link - https://srvoda1.dbi-services.net:7093/docs/cmtrn/issues-with-oda-odacli.html#GUID-F1385628-9F87-4FEF-8D27-289A3ED459EC
  2. Error in restore node process in Data Preserving Reprovisioning
      Abstract - In the Data Preserving Reprovisioning process, during node restore, an error may be encountered.
      Link - https://srvoda1.dbi-services.net:7093/docs/cmtrn/issues-with-oda-odacli.html#GUID-75D52887-D425-4753-AF44-EFAB5C148873
  3. Managing Backup, Restore, and Recovery on a DB System in a KVM Deployment
      Abstract - Understand the backup, restore, and recovery operations supported on a DB system in a KVM deployment.
      Link - https://srvoda1.dbi-services.net:7093/docs/cmtxp/managing-oracle-database-appliance-kvm-deployment1.html#GUID-7318F4D7-4CB8-486C-9DC7-A7490925B866
  4. Backup, Restore and Recover Databases
      Abstract - Review backup, restore, and recovery operations for your databases.
      Link - https://srvoda1.dbi-services.net:7093/docs/cmtxp/backup-recover-restore.html#GUID-032C43EC-20B9-4036-ADA9-7631EEBBFEF6
  5. Using the CLI to Backup, Restore, and Recover
      Abstract - Use the command-line interface to backup, restore, and recover databases.
      Link - https://srvoda1.dbi-services.net:7093/docs/cmtxp/backup-recover-restore.html#GUID-54F9A4A6-59B8-4A18-BE41-1CCB9096E2C5

NOTE: For additional details such as file name and line numbers of error logs, please refer to /opt/oracle/dcs/da/da_repo/924d139e-50cc-4893-ab0a-6acb7e7eeb9c.json

There is a lot of errors, but hard to find what is the source problem.

Let’s have a look at the logfile of the GI setup:

vi /u01/app/oraInventory/logs/GridSetupActions2025-03-05_09-09-40AM/gridSetupActions2025-03-05_09-09-40AM.out
…
Resolution failed, cached value is left untouched for variable
…
No ECDSA host key is known for srvoda1 and you have requested strict checking
…

Could it be a DNS issue? Regarding the network settings, I’m pretty sure that IP, netmask and gateway are OK.

Identify and solve the problem by editing the json file

Like the create-appliance, the restore-node -g will use a json file restored previously with the update-repository of the server archive.

Let’s compare my own backup files and the json file used by restore-node -g:

cat /mnt/backup/ODA_backup/backup_ODA_srvoda1_20250312_0820/resolv.conf
search dbi-services.net
nameserver 10.1.2.127
nameserver 10.1.2.128

cat /opt/oracle/oak/restore/metadata/provisionInstance.json  | grep dns
    "dnsServers" : [ "10.4.110.4", "10.4.110.5" ],

OK, DNS servers have been changed manually in the resolv.conf. They are no more correct in the json file.

As the restore-node -g failed, a cleanup of the system is mandatory:

/opt/oracle/oak/onecmd/cleanup.pl
INFO: Log file is /opt/oracle/oak/log/srvoda1/cleanup/cleanup_2025-03-05_10-40-06.log
INFO: Log file is /opt/oracle/oak/log/srvoda1/cleanup/dcsemu_diag_precleanup_2025-03-05_10-40-06.log

INFO: Platform is 'BM'
INFO: *******************************************************************
INFO: ** Starting process to cleanup provisioned host srvoda1          **
INFO: *******************************************************************
WARNING: DPR environment detected. DPR specific cleanup involves
WARNING: deconfiguring the ODA software stack without touching ASM
WARNING: storage to allow rerunning of the 'odacli restore-node -g'
WARNING: command. If regular cleanup(which erases ASM disk headers)
WARNING: is intended, rerun cleanup.pl with '-nodpr' option.
WARNING: If Multi-User Access is enabled, use '-omausers' option to
WARNING: delete the custom users created during the previous run.
Do you want to continue (yes/no) : yes
INFO:
Running cleanup will delete Grid User - 'grid' and
INFO: DB user - 'oracle' and also the
INFO: groups 'oinstall,dba,asmadmin,asmoper,asmdba'
INFO: nodes will be rebooted
Do you want to continue (yes/no) : yes

INFO: /u01/app/19.24.0.0/grid/bin/crsctl.bin

INFO: *************************************
INFO: ** Checking for GI bits presence
INFO: *************************************
INFO: GI bits /u01/app/19.24.0.0/grid found on system under /u01/app directory...

INFO: *************************************
INFO: ** DPR Cleanup
INFO: *************************************
INFO: Nothing to do.
SUCCESS: DPR cleanup actions completed.
INFO: Attempting to stop DCS agent on local node

INFO: *************************************
INFO: ** Executing AFD cleanup commands
INFO: *************************************
INFO: *************************************
INFO: ** Cleaning Oracle HAMI for ODA
INFO: *************************************
INFO: ** - Oracle HAMI for ODA - ensembles cleaned successfully
INFO: ** - Oracle HAMI for ODA - users cleaned successfully
INFO: *************************************
INFO: ** Executing stack deinstall commands
INFO: *************************************
INFO: *************************************
INFO: ** Removing IPC objects
INFO: *************************************
INFO: Cleaning up IDM configurations...
Deleting directory INFO: *************************************
 
INFO: ** Cleaning miscellaneous components:
INFO: *************************************
INFO: ** - reset limits.conf
INFO: ** - delete users
INFO: ** - delete groups
INFO: ** - hostname, gateway and hosts reset commands
INFO: ** - dcs cleanup and orphan files removal commands
INFO: Attempting to clean MySQL tables on local node

INFO: Cleaning up network bridges
INFO: default net is: pubnet
INFO: Cleaning up network bridges: pubnet on btbond1
INFO: Reset public interface: pubnet
INFO: Cleaning up network bridges: privasm on priv0.100
INFO: remove VLAN config: /etc/sysconfig/network-scripts/ifcfg-priv0.100
INFO: BaseDBCC cleanup - skip
INFO: *************************************
INFO: ** Removing KVM files
INFO: *************************************
INFO: *************************************
INFO: ** Removing BM CPU Pool files
INFO: *************************************
INFO: ** - networking cleaning commands
INFO: ** - UTC reset commands
INFO: *************************************
INFO: ** Removing Oracle AHF RPM
INFO: *************************************
INFO: Oracle AHF RPM is installed as : oracle-ahf-2405000-20240715121646.x86_64
INFO: Uninstalling Oracle AHF RPM
INFO: Oracle AHF RPM uninstalled successfully
INFO: Oracle AHF RPM is installed as : oracle-ahf-2405000-20240715121646.x86_64
INFO: Delete directory clones.local (if existing)...
INFO: Cleaning up ACFS mounts...
INFO: Reset password for 'root' to default value
INFO: Executing 
INFO: Removing SSH keys on srvoda1

INFO: Rebooting the system via ...
INFO: Executing 

INFO: Cleanup was successful
INFO: Log file is /opt/oracle/oak/log/srvoda1/cleanup/cleanup_2025-03-05_10-40-06.log

WARNING: After system reboot, please re-run "odacli update-repository" for GI/DB clones,
WARNING: before running "odacli restore-node -g".

Once the server has rebooted, let’s register again the GI and server archive files:

odacli update-repository -f /mnt/backup/ODA_backup/odacli-dcs-19.24.0.0.0-240724-GI-19.24.0.0.zip
odacli update-repository -f /mnt/backup/ODA_backup/serverarchive_srvoda1.zip

Let’s change the DNS in the provisionInstance.json and do the restore-node -g:

sed -i 's/10.4.110.4/10.1.2.127/g' /opt/oracle/oak/restore/metadata/provisionInstance.json
sed -i 's/10.4.110.5/10.1.2.128/g' /opt/oracle/oak/restore/metadata/provisionInstance.json

cat /opt/oracle/oak/restore/metadata/provisionInstance.json  | grep dns
    "dnsServers" : [ "10.1.2.127", "10.1.2.128" ],

odacli restore-node -g
...

odacli describe-job -i "f2e14691-1fc4-4b8d-9186-cbb55a69c5dd"
Job details
----------------------------------------------------------------
                     ID:  f2e14691-1fc4-4b8d-9186-cbb55a69c5dd
            Description:  Restore node service - GI
                 Status:  Success
                Created:  March 5, 2025 5:33:33 PM CET
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ---------
Restore node service creation            March 5, 2025 5:33:40 PM CET             March 5, 2025 5:53:33 PM CET             Success
Validate absence of Interconnect         March 5, 2025 5:33:40 PM CET             March 5, 2025 5:33:41 PM CET             Success
network configuration file
Setting up Network                       March 5, 2025 5:33:42 PM CET             March 5, 2025 5:33:42 PM CET             Success
Restart network interface pubnet         March 5, 2025 5:33:42 PM CET             March 5, 2025 5:33:48 PM CET             Success
Setting up Vlan                          March 5, 2025 5:33:48 PM CET             March 5, 2025 5:33:49 PM CET             Success
Restart network interface priv0.100      March 5, 2025 5:33:49 PM CET             March 5, 2025 5:33:50 PM CET             Success
Restart network interface privasm        March 5, 2025 5:33:50 PM CET             March 5, 2025 5:33:50 PM CET             Success
Setting up Network                       March 5, 2025 5:33:51 PM CET             March 5, 2025 5:33:51 PM CET             Success
Restart network interface privasm        March 5, 2025 5:33:51 PM CET             March 5, 2025 5:33:56 PM CET             Success
Network update                           March 5, 2025 5:33:56 PM CET             March 5, 2025 5:34:09 PM CET             Success
Updating network                         March 5, 2025 5:33:56 PM CET             March 5, 2025 5:34:09 PM CET             Success
Setting up Network                       March 5, 2025 5:33:56 PM CET             March 5, 2025 5:33:56 PM CET             Success
Restart network interface btbond1        March 5, 2025 5:33:56 PM CET             March 5, 2025 5:34:00 PM CET             Success
Restart network interface btbond1        March 5, 2025 5:34:00 PM CET             March 5, 2025 5:34:03 PM CET             Success
Restart network interface pubnet         March 5, 2025 5:34:03 PM CET             March 5, 2025 5:34:09 PM CET             Success
Validate availability of pubnet          March 5, 2025 5:34:09 PM CET             March 5, 2025 5:34:09 PM CET             Success
OS usergroup 'asmdba' creation           March 5, 2025 5:34:09 PM CET             March 5, 2025 5:34:09 PM CET             Success
OS usergroup 'asmoper' creation          March 5, 2025 5:34:09 PM CET             March 5, 2025 5:34:09 PM CET             Success
OS usergroup 'asmadmin' creation         March 5, 2025 5:34:09 PM CET             March 5, 2025 5:34:09 PM CET             Success
OS usergroup 'dba' creation              March 5, 2025 5:34:09 PM CET             March 5, 2025 5:34:10 PM CET             Success
OS usergroup 'dbaoper' creation          March 5, 2025 5:34:10 PM CET             March 5, 2025 5:34:10 PM CET             Success
OS usergroup 'oinstall' creation         March 5, 2025 5:34:10 PM CET             March 5, 2025 5:34:10 PM CET             Success
OS user 'grid' creation                  March 5, 2025 5:34:10 PM CET             March 5, 2025 5:34:10 PM CET             Success
OS user 'oracle' creation                March 5, 2025 5:34:10 PM CET             March 5, 2025 5:34:11 PM CET             Success
Default backup policy creation           March 5, 2025 5:34:11 PM CET             March 5, 2025 5:34:11 PM CET             Success
Backup Config name validation            March 5, 2025 5:34:11 PM CET             March 5, 2025 5:34:11 PM CET             Success
Backup config metadata persist           March 5, 2025 5:34:11 PM CET             March 5, 2025 5:34:11 PM CET             Success
Grant permission to RHP files            March 5, 2025 5:34:11 PM CET             March 5, 2025 5:34:11 PM CET             Success
Add SYSNAME in Env                       March 5, 2025 5:34:11 PM CET             March 5, 2025 5:34:11 PM CET             Success
Install oracle-ahf                       March 5, 2025 5:34:11 PM CET             March 5, 2025 5:35:54 PM CET             Success
Stop DCS Admin                           March 5, 2025 5:35:56 PM CET             March 5, 2025 5:35:56 PM CET             Success
Generate mTLS certificates               March 5, 2025 5:35:56 PM CET             March 5, 2025 5:35:58 PM CET             Success
Exporting Public Keys                    March 5, 2025 5:35:58 PM CET             March 5, 2025 5:35:59 PM CET             Success
Creating Trust Store                     March 5, 2025 5:35:59 PM CET             March 5, 2025 5:36:02 PM CET             Success
Update config files                      March 5, 2025 5:36:02 PM CET             March 5, 2025 5:36:02 PM CET             Success
Restart DCS Admin                        March 5, 2025 5:36:02 PM CET             March 5, 2025 5:36:23 PM CET             Success
Unzipping storage configuration files    March 5, 2025 5:36:23 PM CET             March 5, 2025 5:36:23 PM CET             Success
Reloading multipath devices              March 5, 2025 5:36:23 PM CET             March 5, 2025 5:36:24 PM CET             Success
Restart oakd                             March 5, 2025 5:36:24 PM CET             March 5, 2025 5:36:34 PM CET             Success
Restart oakd                             March 5, 2025 5:37:45 PM CET             March 5, 2025 5:37:55 PM CET             Success
Restore Quorum Disks                     March 5, 2025 5:37:55 PM CET             March 5, 2025 5:37:56 PM CET             Success
Creating GI home directories             March 5, 2025 5:37:56 PM CET             March 5, 2025 5:37:56 PM CET             Success
Extract GI clone                         March 5, 2025 5:37:56 PM CET             March 5, 2025 5:39:07 PM CET             Success
Creating wallet for Root User            March 5, 2025 5:39:07 PM CET             March 5, 2025 5:39:15 PM CET             Success
Creating wallet for ASM Client           March 5, 2025 5:39:15 PM CET             March 5, 2025 5:39:18 PM CET             Success
Grid stack creation                      March 5, 2025 5:39:18 PM CET             March 5, 2025 5:49:35 PM CET             Success
GI Restore with RHP                      March 5, 2025 5:39:18 PM CET             March 5, 2025 5:46:26 PM CET             Success
Updating GIHome version                  March 5, 2025 5:46:28 PM CET             March 5, 2025 5:46:31 PM CET             Success
Restarting Clusterware                   March 5, 2025 5:46:32 PM CET             March 5, 2025 5:49:35 PM CET             Success
Post cluster OAKD configuration          March 5, 2025 5:49:35 PM CET             March 5, 2025 5:50:30 PM CET             Success
Mounting disk group DATA                 March 5, 2025 5:50:30 PM CET             March 5, 2025 5:50:31 PM CET             Success
Mounting disk group RECO                 March 5, 2025 5:50:38 PM CET             March 5, 2025 5:50:45 PM CET             Success
Setting ACL for disk groups              March 5, 2025 5:50:53 PM CET             March 5, 2025 5:50:55 PM CET             Success
Register Scan and Vips to Public Network March 5, 2025 5:50:55 PM CET             March 5, 2025 5:50:57 PM CET             Success
Adding Volume DUMPS to Clusterware       March 5, 2025 5:51:10 PM CET             March 5, 2025 5:51:13 PM CET             Success
Adding Volume ACFSCLONE to Clusterware   March 5, 2025 5:51:13 PM CET             March 5, 2025 5:51:15 PM CET             Success
Adding Volume ODABASE_N0 to Clusterware  March 5, 2025 5:51:15 PM CET             March 5, 2025 5:51:18 PM CET             Success
Adding Volume COMMONSTORE to Clusterware March 5, 2025 5:51:18 PM CET             March 5, 2025 5:51:20 PM CET             Success
Adding Volume ORAHOME_SH to Clusterware  March 5, 2025 5:51:20 PM CET             March 5, 2025 5:51:23 PM CET             Success
Enabling Volume(s)                       March 5, 2025 5:51:23 PM CET             March 5, 2025 5:52:26 PM CET             Success
Discover ACFS clones config              March 5, 2025 5:53:18 PM CET             March 5, 2025 5:53:27 PM CET             Success
Configure export clones resource         March 5, 2025 5:53:26 PM CET             March 5, 2025 5:53:27 PM CET             Success
Discover DbHomes ACFS config             March 5, 2025 5:53:27 PM CET             March 5, 2025 5:53:30 PM CET             Success
Discover OraHomeStorage volumes          March 5, 2025 5:53:27 PM CET             March 5, 2025 5:53:30 PM CET             Success
Setting up Hugepages                     March 5, 2025 5:53:30 PM CET             March 5, 2025 5:53:30 PM CET             Success
Provisioning service creation            March 5, 2025 5:53:32 PM CET             March 5, 2025 5:53:32 PM CET             Success
Persist new agent state entry            March 5, 2025 5:53:32 PM CET             March 5, 2025 5:53:32 PM CET             Success
Persist new agent state entry            March 5, 2025 5:53:32 PM CET             March 5, 2025 5:53:32 PM CET             Success
Restart DCS Agent                        March 5, 2025 5:53:32 PM CET             March 5, 2025 5:53:33 PM CET             Success

This time it worked fine. Next step will be the restore-node -d for restoring the databases.

The backup script I use before patching an ODA

Troubleshooting is easier if you can have a look at configuration files that were in use prior reimaging. Here is a script I’ve been using for years before patching or reimaging an ODA. I would recommend making your own script based on mine according to your specific configuration:

vi /mnt/backup/prepatch_backup.sh
# Backup important files before patching
export BKPPATH=/mnt/backup/ODA_backup/backup_ODA_`hostname`_`date +"%Y%m%d_%H%M"`
echo "Backing up to " $BKPPATH
mkdir -p $BKPPATH
odacli list-databases > $BKPPATH/list-databases.txt
ps -ef | grep pmon | grep -v ASM | grep -v APX | grep -v grep | cut -c 58- | sort > $BKPPATH/running-instances.txt
odacli list-dbhomes > $BKPPATH/list-dbhomes.txt
odacli list-dbsystems > $BKPPATH/list-dbsystems.txt
odacli list-vms > $BKPPATH/list-vms.txt
crontab -u oracle -l  > $BKPPATH/crontab-oracle.txt
crontab -u grid -l  > $BKPPATH/crontab-grid.txt
crontab -l  > $BKPPATH/crontab-root.txt

cat /etc/fstab >  $BKPPATH/fstab.txt
cat /etc/oratab >  $BKPPATH/oratab.txt
cat /etc/sysconfig/network >  $BKPPATH/etc-sysconfig-network.txt
cat /etc/hosts  >  $BKPPATH/hosts
cat /etc/resolv.conf  >  $BKPPATH/resolv.conf
cp /etc/krb5.conf  $BKPPATH/
cp /etc/krb5.keytab  $BKPPATH/
mkdir $BKPPATH/network-scripts
cp  /etc/sysconfig/network-scripts/ifcfg*  $BKPPATH/network-scripts/
odacli describe-system > $BKPPATH/describe-system.txt
odacli  describe-component >  $BKPPATH/describe-component.txt
HISTFILE=~/.bash_history
set -o history
history > $BKPPATH/history-root.txt
cp /home/oracle/.bash_history $BKPPATH/history-oracle.txt
df -h >  $BKPPATH/filesystems-status.txt

for a in `odacli list-dbhomes -j | grep dbHomeLocation | awk -F '"' '{print $4}' | sort` ; do mkdir -p $BKPPATH/$a/network/admin/ ; cp $a/network/admin/tnsnames.ora $BKPPATH/$a/network/admin/; cp $a/network/admin/sqlnet.ora $BKPPATH/$a/network/admin/; done
for a in `odacli list-dbhomes -j | grep dbHomeLocation | awk -F '"' '{print $4}' | sort` ; do mkdir -p $BKPPATH/$a/owm/ ; cp -r $a/owm/* $BKPPATH/$a/owm/; done
cp `ps -ef | grep -v grep | grep LISTENER | awk -F ' ' '{print $8}' | awk -F 'bin' '{print $1}'`network/admin/listener.ora $BKPPATH/gridhome-listener.ora
cp `ps -ef | grep -v grep | grep LISTENER | awk -F ' ' '{print $8}' | awk -F 'bin' '{print $1}'`/network/admin/sqlnet.ora $BKPPATH/gridhome-sqlnet.ora

tar czf $BKPPATH/u01-app-oracle-admin.tgz /u01/app/oracle/admin/
tar czf $BKPPATH/u01-app-oracle-local.tgz /u01/app/oracle/local/
tar czf $BKPPATH/home.tgz /home/
cp /etc/passwd $BKPPATH/
cp /etc/group $BKPPATH/

echo "End"
echo "Backup files size:"
du -hs  $BKPPATH
echo "Backup files content:"
ls -lrt  $BKPPATH

Conclusion

I would recommend these 3 rules to save hours of troubleshooting:

as mentioned in the ODA documentation: never change networks parameters manually (IP, gateway, hostname, DNS, bonding mode, …)
document every manual change you make on your ODA (additional tools setup, specific settings, …)
do an extensive backup of all configuration files before doing a patching or a DPR (use the script provided in the previous chapter for creating yours)

L’article PRGH-1030 when doing restore-node -g on ODA est apparu en premier sur dbi Blog.

Troubleshooting Oracle Data Guard

Jérôme Dubar — Fri, 21 Feb 2025 10:06:03 +0000

Introduction

Oracle Data Guard is a brilliant component of Oracle Database Enterprise Edition. On its own, it’s a strong argument in favor of Enterprise Edition. This is because Data Guard is a proven Disaster Recovery solution: it’s integrated, reliable and well known by DBAs. And it comes at no additional cost if you don’t need the Active Guard feature. Several days ago, I helped colleagues of mine troubleshooting a Data Guard configuration: let’s summarize how I would proceed to make my standby database back to sync if I have a problem. In this example, I consider having a simple environment with one primary and one standby in Maximum Availability. If you have multiple standby databases, the tasks are the same.

How to check your Data Guard setup?

The easiest way to check your Data Guard setup is by using the Data Guard CLI called the “broker” from one of the servers. The SHOW CONFIGURATION LAG will display 2 key things: your actual configuration (who is primary, who is standby) and the lag between the databases. The lag should be 0 in most cases:

. oraenv <<< DBTEST
dgmgrl / "show configuration lag;"

DGMGRL for Linux: Release 19.0.0.0.0 - Production on Wed Feb 19 15:06:17 2025
Version 19.20.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected to "DBTEST_IN"
Connected as SYSDG.

Configuration - dtg

  Protection Mode: MaxAvailability
  Members:
  dbtest_in - Primary database
    dbtest_cr - Physical standby database
                Transport Lag:      0 seconds (computed 1 second ago)
                Apply Lag:          0 seconds (computed 1 second ago)

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 26 seconds ago)

If the lag is bigger than a couple of seconds or if you have errors in your configuration, you will need to do some troubleshooting.

This is an example of a Data Guard configuration not working correctly:

Configuration - dtg

  Protection Mode: MaxAvailability
  Members:
  dbtest_in - Primary database
    Error: ORA-16810: multiple errors or warnings detected for the member

    dbtest_cr - Physical standby database
      Warning: ORA-16854: apply lag could not be determined

Fast-Start Failover:  Disabled

Configuration Status:
ERROR   (status updated 56 seconds ago)

Check the Fast Recovery Area on your standby database

A common problem you may have on a standby database is the Fast Recovery Area (FRA) being full. You probably know that a standby database will apply the changes from the primary without waiting for the archivelog to be shipped, but the archivelog is shipped anyway to the standby. This is because there is no guarantee that the standby database is always up and running, so archivelogs must be transported to the other site for later apply if needed. Another thing is that you probably enabled Flashback Database on both databases, and archivelogs are required for a Flashback Database operation. These shipped archivelogs will naturally land in the FRA, and unless you configured a deletion policy, they will never be deleted. As you probably don’t do backups on the standby database, nothing could flag these archivelogs as reclaimable (meaning that they are now useless and can be deleted). If your FRA is quite big, you may discover this problem several weeks or months after the initial setup.

Check the FRA usage on your standby with this query:

select  sum(PERCENT_SPACE_USED-PERCENT_SPACE_RECLAIMABLE) "Real FRA usage %" from v$flash_recovery_area_usage;
Real FRA usage %
----------------
           32.86

If the FRA is almost full, you can remove older archivelogs, for example those older than 2 days if your standby has a 1-day lag:

rman target /
delete force noprompt archivelog all completed before 'sysdate-2';
exit;

Then check again the FRA and the lag of your Data Guard setup.

Check standby_file_management parameter

Another thing that can break your sync is the standby_file_management parameter having an incorrect value. In most cases, it must be set to AUTOMATIC: it means that any file created on the primary will be created on the standby. It’s the way it is supposed to work:

show parameter standby_file_management
NAME				     TYPE	 VALUE
------------------------------------ ----------- ------------------------------
standby_file_management 	     string	 AUTO

If standby_file_management is set to MANUAL, your standby will not be in sync anymore as soon as a new datafile is created on the primary. You will need to manually create the file on the standby to continue the sync. Not very convenient.

The MANUAL mode exists because some older configurations had different filesystems on primary and on standby and didn’t use OMF, meaning that standby database cannot guess where to put the new file.

Both primary and standby databases must have this parameter set to AUTO nowadays.

Cross test connexions

Your Data Guard setup can only work if your databases are able to communicate together. If you are not so sure if something changed on your network, for example a new firewall rule, check your connexions from both servers using the SYS account.

From your primary:

sqlplus sys@DBTEST_CR as sysdba
sqlplus sys@DBTEST_IN as sysdba

From your standby:

sqlplus sys@DBTEST_CR as sysdba
sqlplus sys@DBTEST_IN as sysdba

Check password file

When your standby database is MOUNTED, the only way to authenticate the SYS user is by using the password file. If you changed the SYS password on the primary, it will be changed inside the database (and replicated) as well as in the local password file, but the password file on the standby site won’t be updated. You must then copy the password file from the primary database to the standby database. Copy is done with a scp command, for example from my primary server:


srvctl config database -db DBTEST_IN | grep Password
Password file: /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4/dbs/orapwDBTEST

scp `srvctl config database -db OP1 | grep Password | awk '{print $3;}'` oracle@oda-x11-cr:`srvctl config database -db OP1 | grep Password | awk '{print $3;}'`

A restart of the standby database may be needed.

Check alert_DBTEST.log and drcDBTEST.log on both servers

Never miss an error reported in the alert_DBTEST.log on both sides. I would recommend disabling the Data Guard configuration, doing a tail -f on both alert_DBTEST.log files, and enabling back the configuration:

. oraenv <<< DBTEST
dgmgrl / "disable configuration;"
sleep 60
dgmgrl / "enable configuration;"

There are also dedicated trace files for Data Guard, at the same place as alert_DBTEST.log: drcDBTEST.log. You may find additional information for troubleshooting your configuration in these files.

Remove and recreate the configuration

Data Guard configuration is just a couple of parameters stored in a file on both sides. It’s easy to drop and create again this configuration without actually rebuilding the standby database. If you want to make sure that nothing survives from your old configuration, just remove the broker files before creating the configuration again.

From the primary:

dgmgrl sys
edit configuration set protection mode as maxperformance;
remove configuration;
exit;

sqlplus / as sysdba
sho parameter dg_broker_config_file

NAME				     TYPE	 VALUE
------------------------------------ ----------- ------------------------------
dg_broker_config_file1		     string	 /u01/app/odaorahome/oracle/pro
						 duct/19.0.0.0/dbhome_4/dbs/dr1
						 DBTEST_IN.dat
dg_broker_config_file2		     string	 /u01/app/odaorahome/oracle/pro
						 duct/19.0.0.0/dbhome_4/dbs/dr2
						 DBTEST_IN.dat
alter system set dg_broker_start=FALSE;

host rm /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4/dbs/dr1DBTEST_IN.dat
host rm /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4/dbs/dr2DBTEST_IN.dat

exit

From the standby:

sqlplus / as sysdba
sho parameter dg_broker_config_file

NAME				     TYPE	 VALUE
------------------------------------ ----------- ------------------------------
dg_broker_config_file1		     string	 /u01/app/odaorahome/oracle/pro
						 duct/19.0.0.0/dbhome_4/dbs/dr1
						 DBTEST_CR.dat
dg_broker_config_file2		     string	 /u01/app/odaorahome/oracle/pro
						 duct/19.0.0.0/dbhome_4/dbs/dr2
						 DBTEST_CR.dat
alter system set dg_broker_start=FALSE;

host rm /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4/dbs/dr1DBTEST_CR.dat
host rm /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4/dbs/dr2DBTEST_CR.dat

alter system set dg_broker_start=TRUE;
exit

From the primary:

sqlplus / as sysdba
alter system set dg_broker_start=TRUE;
exit;

dgmgrl sys
create configuration DTG as primary database is 'DBTEST_IN' connect identifier is 'DBTEST_IN';
add database 'DBTEST_CR' as connect identifier is 'DBTEST_CR';
enable configuration;
edit database 'DBTEST_CR' set property LogXptMode='SYNC';
edit database 'DBTEST_IN' set property LogXptMode='SYNC';
edit database 'DBTEST_CR' set property StandbyFileManagement='AUTO';
edit database 'DBTEST_IN' set property StandbyFileManagement='AUTO';
EDIT DATABASE 'DBTEST_CR' SET PROPERTY 'ArchiveLagTarget'=1200;
EDIT DATABASE 'DBTEST_IN' SET PROPERTY 'ArchiveLagTarget'=1200;
Edit database 'DBTEST_CR' set property StaticConnectIdentifier='(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oda-x11-cr)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=DBTEST_CR)(INSTANCE_NAME=DBTEST)(SERVER=DEDICATED)))';
Edit database 'DBTEST_IN' set property StaticConnectIdentifier='(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oda-x11-in)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=DBTEST_IN)(INSTANCE_NAME=DBTEST)(SERVER=DEDICATED)))';
edit configuration set protection mode as maxavailability;
show configuration lag;
exit;

Note that the StaticConnectIdentifier property is only mandatory when using a port different than 1521.

Recover standby database from service

If you still struggle to get your standby back in sync, because too many archivelogs are missing or because the archivelogs are not on the primary site anymore, you can use this nice RMAN command on your standby:

sqlplus / as sysdba
alter system set dg_broker_start=false;
exit;

srvctl stop database -db DBTEST_CR
sleep 10
srvctl start database -db DBTEST_CR -o mount

rman target /
recover database from service 'DBTEST_IN';
exit;

sqlplus / as sysdba
alter system set dg_broker_start=true;
exit;

This RECOVER DATABASE FROM SERVICE will do an incremental backup on the primary to recover the standby without needing the missing archivelogs. It’s convenient and much faster than rebuilding the standby from scratch.

Check SCN

In the good old days of Data Guard on Oracle 9i, the broker didn’t exist and you had to configure everything yourself. At this time, I used to have a look at the SCN on both databases for monitoring the lag. Nothing changed regarding the SCN: on a primary, you will never see the same SCN each time you query its value. This is because the query itself will increase the SCN by 1, as well as other background queries are running:

select current_scn from v$database;
CURRENT_SCN
-----------
  271650667

select current_scn from v$database;
CURRENT_SCN
-----------
  271650674

select current_scn from v$database;
CURRENT_SCN
-----------
  271650675

select current_scn from v$database;
CURRENT_SCN
-----------
  271650678

On a standby database, the SCN can only increase if changes are pushed by a primary. And for sure, the SCN will always be lower than the one on the primary. If your standby database is not opened, meaning that you don’t have the Active Guard option, you will query the same SCN for a couple of minutes, and you will see big jumps in figures from time to time:

select current_scn from v$database;
CURRENT_SCN
-----------
  271650664

select current_scn from v$database;
CURRENT_SCN
-----------
  271650664

select current_scn from v$database;
CURRENT_SCN
-----------
  271651042

select current_scn from v$database;
CURRENT_SCN
-----------
  271651042

Conclusion

With these few tips, troubleshooting Data Guard might be easier now.

L’article Troubleshooting Oracle Data Guard est apparu en premier sur dbi Blog.