Marc Wagner, auteur/autrice sur dbi Blog https://www.dbi-services.com/blog/author/marc-wagner/ Fri, 22 May 2026 13:34:48 +0000 en-US hourly 1 https://www.dbi-services.com/blog/wp-content/uploads/sites/2/2025/05/cropped-favicon_512x512px-min-32x32.png Marc Wagner, auteur/autrice sur dbi Blog https://www.dbi-services.com/blog/author/marc-wagner/ 32 32 Oracle Database 26ai Client and SQLNET.EXPIRE_TIME https://www.dbi-services.com/blog/oracle-database-26ai-client-and-sqlnet-expire_time/ https://www.dbi-services.com/blog/oracle-database-26ai-client-and-sqlnet-expire_time/#respond Fri, 22 May 2026 13:34:46 +0000 https://www.dbi-services.com/blog/?p=44747 We have been facing one issue at one of our customer where the Oracle Client connections remained opened for days blocking some avaloq JobNetz. We have been doing some tests and we could fortunately find a solution resolving the problem thanks to Oracle Database 26ai supporting now SQLNET.EXPIRE_TIME on the client side. Through this blog, […]

L’article Oracle Database 26ai Client and SQLNET.EXPIRE_TIME est apparu en premier sur dbi Blog.

]]> We have been facing one issue at one of our customer where the Oracle Client connections remained opened for days blocking some avaloq JobNetz. We have been doing some tests and we could fortunately find a solution resolving the problem thanks to Oracle Database 26ai supporting now SQLNET.EXPIRE_TIME on the client side. Through this blog, I would like to share with you the problem and then the tests that have been performed helping us to conclude to a solution.

Environment and problem description

At our customer environment, client connection run from the HelperVM does not establish database connections directly to the database listener. The connection goes through the Network Load Balancer, so called NLB, and the Oracle Connection Manager, so called CMAN.

The diagram below describes the database connection establishment process.

This is how it works.

  • 1 – Client seeks for connection details (ideally, get the connection details from Oracle Directory Service)
  • 2 – Client connects to Network Load Balancer
  • 3 – Network Load Balancer “forwards” the request to Oracle Connection MANager using Virtual IP
  • 4 – Oracle Connection MANager acts as a rule-based firewall and ensure the database target service is running on the “white listed targets (next_hop)
  • 5 – Oracle Database establish connectivity upon credential validate (Oracle listener acts in between). The listener hands the connection over to the Oracle CMAN gateway process, which passes data back and forth between the client and the db-server and collects statistics.

We could observed per reverse engineering  technique that the TCP connection established between the Oracle client and Oracle CMAN works upon Network Load Balancer Virtual IP.

The Network Load Balancer needs for Session persistence “statefullnes” to be enabled. This means that once the connection is established, the NLB “remembers” established connections and fails them over in case of planned downtime.

We have been facing some broken connectivity issue. Checking Linux socket connection with linux ss command (# ss -nop) we could see TCP connection hungs between client and NLB virtual IP (CNAME DNS entry). On CMAN and DB-Server side the connection were already cleaned up as per Dead Connection Detection configuration. We can see in the diagram that EXPIRE_TIME is setup with a value of 10 minutes on the CMAN side and the listener configuration from the VM Cluster database.

The connection was still opened on the client side because:

  • The client was still waiting on a result which would never come
  • TCP connection to the NLB Virtual IP was still existing albeit closed with the CMAN and database listener
  • TCP connection to the NLB would remain alive for days

The problem is that by default Oracle client does not enable TCP Keepalive, which is an expected behavior. Oracle expects the keepalive to be set on the server side. The “Dead Connection Detection” will then be enforced for all clients.

TCP Keepalive should then be managed in our case on the client side. And we are currently running Oracle 19c Client.

EXPIRE_TIME handled on Oracle 19c Client

Oracle 19c client does not come with SQLNET.EXPIRE_TIME aka “Oracle dead connection detection”, unless hacked over connection string hidden (unsupported) parameter ENABLE_BROKEN. 

See following blog from a one of my former colleagues:

sqlnet-expire_time and enablebroken

And what about Oracle 26ai Client? Let’s do some test…

Installation of Oracle 26ai Client

On the lab, I will use the VM called bastion to act as the client. The bastion has already an Oracle 19c Client installed. I’m going to installed new Oracle 26ai Client on it.

First we need to download the client version, which can be done from the following website:

https://www.oracle.com/database/technologies/oracle26ai-linux-downloads.html

I will install Oracle 26ai Client version in /opt/oracle.

[root@bastion oracle]# pwd
/opt/oracle

[root@bastion oracle]# ls -ld client*
drwxr-xr-x. 52 oracle oinstall 4096 Jun  6  2024 client19c
drwxr-xr-x. 47 oracle oinstall 4096 Nov 11  2025 client_21c
[root@bastion oracle]#

I will first unzip the downloaded oracle zip file.

[oracle@bastion oracle]$ pwd
/opt/oracle

[oracle@bastion oracle]$ unzip -q LINUX.X64_2326100_client.zip

I will then rename the client installation directory:

[oracle@bastion oracle]$ ls -ld client*
drwxr-xr-x.  5 oracle oinstall   90 Jan 17 13:59 client
drwxr-xr-x. 52 oracle oinstall 4096 Jun  6  2024 client19c
drwxr-xr-x. 47 oracle oinstall 4096 Nov 11  2025 client_21c

[oracle@bastion oracle]$ mv client client26ai
[oracle@bastion oracle]$

[oracle@bastion oracle]$ ls -ld client*
drwxr-xr-x. 52 oracle oinstall 4096 Jun  6  2024 client19c
drwxr-xr-x. 47 oracle oinstall 4096 Nov 11  2025 client_21c
drwxr-xr-x.  5 oracle oinstall   90 Jan 17 13:59 client26ai
[oracle@bastion oracle]$

I will prepare the response file for the command line installation.

[oracle@bastion client26ai]$ cp -p response/client_install.rsp response/client_install_custom.rsp

[oracle@bastion client26ai]$ vi response/client_install_custom.rsp

[oracle@bastion client26ai]$ diff response/client_install.rsp response/client_install_custom.rsp
22c22
 UNIX_GROUP_NAME=oinstall
26c26
 INVENTORY_LOCATION=/opt/oracle/oraInventory
30c30
 ORACLE_HOME=/opt/oracle/client26ai
34c34
 ORACLE_BASE=/opt/oracle
48c48
 oracle.install.client.installType=Administrator
[oracle@bastion client26ai]$

And I will run the Oracle 26ai Client installation.

[oracle@bastion client26ai]$ pwd
/opt/oracle/client26ai

[oracle@bastion client26ai]$ ls -ltrh
total 24K
-rwxrwx---.  1 oracle oinstall  500 Feb  6  2013 welcome.html
-rwxr-xr-x.  1 oracle oinstall 8.7K Jan 17 12:38 runInstaller
drwxr-xr-x.  4 oracle oinstall 4.0K Jan 17 12:38 install
drwxr-xr-x. 15 oracle oinstall 4.0K Jan 17 13:37 stage
drwxr-xr-x.  2 oracle oinstall   82 May 21 16:13 response

[oracle@bastion client26ai]$ ./runInstaller -silent -responseFile /opt/oracle/client26ai/response/client_install_custom.rsp
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 415 MB.   Actual 5025 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 4095 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2026-05-21_04-16-17PM. Please wait ... [WARNING] [INS-32016] The selected Oracle home contains directories or files.
   ACTION: To start with an empty Oracle home, either remove its contents or specify a different location.
*********************************************
Package: compat-openssl10-1.0.2 (x86_64): This is a prerequisite condition to test whether the package "compat-openssl10-1.0.2 (x86_64)" is available on the system.
Severity: IGNORABLE
Overall status: VERIFICATION_FAILED
Error message: PRVF-7532 : Package "compat-openssl10(x86_64)-1.0.2" is missing on node "bastion"
Cause:  A required package is either not installed or, if the package is a kernel module, is not loaded on the specified node.
Action:  Ensure that the required package is installed and available.
-----------------------------------------------
[WARNING] [INS-13014] Target environment does not meet some optional requirements.
   CAUSE: Some of the optional prerequisites are not met. See logs for details. /opt/oraInventory/logs/installActions2026-05-21_04-16-17PM.log.
   ACTION: Identify the list of failed prerequisite checks from the log: /opt/oraInventory/logs/installActions2026-05-21_04-16-17PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
The response file for this session can be found at:
 /opt/oracle/client26ai/install/response/client_2026-05-21_04-16-17PM.rsp

You can find the log of this install session at:
 /opt/oraInventory/logs/installActions2026-05-21_04-16-17PM.log

The installation of Oracle Client 26ai was successful.
Please check '/opt/oraInventory/logs/silentInstall2026-05-21_04-16-17PM.log' for more details.
Successfully Setup Software with warning(s).
[INS-10115] All configuration tools were previously ran successfully, no further configuration is required.

[oracle@bastion client26ai]$

And the Oracle 26ai Client is now installed.

Prepare target database

I will create a user on the target lab PDB, named TESTZ_TMR_003I, in order to establish sqlplus connection and test the EXPIRE_TIME configuration.

I will create a user test01 and grant the connect permissions.

[oracle@svl-oat ~]$ echo $ORACLE_SID
CDB001I

[oracle@svl-oat ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Thu May 21 14:30:39 2026
Version 19.23.0.0.0

Copyright (c) 1982, 2023, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c EE High Perf Release 19.0.0.0.0 - Production
Version 19.23.0.0.0

SQL> show pdbs

    CON_ID CON_NAME           OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
     2 PDB$SEED           READ ONLY  NO
     3 TESTZ_APP_006I         READ WRITE NO
     4 RCLON_TMR_003I         MOUNTED
     5 RCLON_TMR_002I         MOUNTED
     6 RCLON_TMR_001I         MOUNTED
     7 CLONZ_TMR_002I         MOUNTED
     8 TESTZ_TMR_003I         READ WRITE NO
    10 TESTZ_APP_004I         READ WRITE NO
    11 CLONZ_APP_001I         READ WRITE NO
    12 RCLON_APP_003I         READ WRITE NO

SQL> alter session set container=TESTZ_TMR_003I;

Session altered.

SQL> create user test01 identified by "test_expire";

User created.

SQL> grant connect to test01;

Grant succeeded.

SQL>

Test connection with Oracle 26ai Client

I will first set the ORACLE_HOME variable on the appropriate client directory.

[oracle@bastion client26ai]$ echo $ORACLE_HOME
/opt/oracle/client19c

[oracle@bastion client26ai]$ export ORACLE_HOME=/opt/oracle/client26ai

[oracle@bastion client26ai]$ echo $ORACLE_HOME
/opt/oracle/client26ai
[oracle@bastion client26ai]$

I will update the PATH variable to get tnsping and sqlplus binary from the appropriate Oracle 26ai Client.

[oracle@bastion client26ai]$ echo $PATH
/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/oracle/client19c/bin:/opt/oracle/sqlcl-24.1.0.087.0929//bin

[oracle@bastion client26ai]$ export PATH=/opt/oracle/client26ai/bin

[oracle@bastion client26ai]$ echo $PATH
/opt/oracle/client26ai/bin
[oracle@bastion client26ai]$

I will check that the appropriate tnsping and sqlplus is taken.

[oracle@bastion client26ai]$ which sqlplus
/opt/oracle/client26ai/bin/sqlplus

[oracle@bastion client26ai]$ which tnsping
/opt/oracle/client26ai/bin/tnsping

I checked that the connection to target PDB is working.

[oracle@bastion ~]$ tnsping svl-oat:1521/testz_tmr_003i.db.jewlab.oraclevcn.com

TNS Ping Utility for Linux: Version 23.26.1.0.0 - Production on 21-MAY-2026 16:28:56

Copyright (c) 1997, 2026, Oracle.  All rights reserved.

Used parameter files:
/opt/oracle/client19c/network/admin/sqlnet.ora

Used EZCONNECT adapter to resolve the alias
Attempting to contact (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=testz_tmr_003i.db.jewlab.oraclevcn.com))(ADDRESS=(PROTOCOL=tcp)(HOST=X.X.1.135)(PORT=1521)))
OK (0 msec)
[oracle@bastion ~]$

The TNS_ADMIN used is from the 19c Oracle client directory, which is absolutely not a problem.

[oracle@bastion ~]$ echo $TNS_ADMIN
/opt/oracle/client19c/network/admin
[oracle@bastion ~]$

Test sqlplus connection with Oracle 26ai Client

As I can see on my client side, I do not have any sqlplus connection running right now.

[opc@bastion ~]$ ss -nop | grep 1521
[opc@bastion ~]$

I will generate a sqlplus connection.

[oracle@bastion client26ai]$ sqlplus test01/test_expire@svl-oat:1521/testz_tmr_003i.db.jewlab.oraclevcn.com

SQL*Plus: Release 23.26.1.0.0 - Production on Thu May 21 16:34:23 2026
Version 23.26.1.0.0

Copyright (c) 1982, 2025, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c EE High Perf Release 19.0.0.0.0 - Production
Version 19.23.0.0.0

SQL>

And I can see that I have got a sqlplus connection with no timer/keepalive.

[opc@bastion ~]$ ss -nop | grep 1521
tcp   ESTAB  0      0   X.X.0.89:51404    X.X.1.135:1521
[opc@bastion ~]$

I will now configure Oracle Dead Connection Detection with SQLNET.EXPIRE_TIME parameter set in the client sqlnet.ora with a value of 1 minute.

[oracle@bastion ~]$ cd $TNS_ADMIN

[oracle@bastion admin]$ /usr/bin/grep -i expire sqlnet.ora
[oracle@bastion admin]$ 

[oracle@bastion admin]$ /usr/bin/vi sqlnet.ora

[oracle@bastion admin]$ /usr/bin/grep -i expire sqlnet.ora
SQLNET.EXPIRE_TIME=1
[oracle@bastion admin]$

I will run a new sqlplus connection.

[oracle@bastion admin]$ sqlplus test01/test_expire@svl-oat:1521/testz_tmr_003i.db.jewlab.oraclevcn.com

SQL*Plus: Release 23.26.1.0.0 - Production on Thu May 21 16:41:06 2026
Version 23.26.1.0.0

Copyright (c) 1982, 2025, Oracle.  All rights reserved.

Last Successful login time: Thu May 21 2026 16:34:23 +02:00

Connected to:
Oracle Database 19c EE High Perf Release 19.0.0.0.0 - Production
Version 19.23.0.0.0

SQL>

I can now see that I have got a connection configured with a timer and keep alive remaining of 38s.

[opc@bastion ~]$ ss -nop | grep 1521
tcp   ESTAB  0      0     X.X.0.89:62868    X.X.1.135:1521   timer:(keepalive,38sec,0)
[opc@bastion ~]$

Let’s configure the EXPIRE_TIME with a value of 15 minutes.

[oracle@bastion admin]$ /usr/bin/vi sqlnet.ora

[oracle@bastion admin]$ /usr/bin/grep -i expire sqlnet.ora
SQLNET.EXPIRE_TIME=15

I run a new sqlplus connection.

[oracle@bastion admin]$ sqlplus test01/test_expire@svl-oat:1521/testz_tmr_003i.db.jewlab.oraclevcn.com

SQL*Plus: Release 23.26.1.0.0 - Production on Thu May 21 16:43:25 2026
Version 23.26.1.0.0

Copyright (c) 1982, 2025, Oracle.  All rights reserved.

Last Successful login time: Thu May 21 2026 16:42:29 +02:00

Connected to:
Oracle Database 19c EE High Perf Release 19.0.0.0.0 - Production
Version 19.23.0.0.0

SQL>

And I now have got a connection configured with a timer and keep alive remaining of 14min.

[opc@bastion ~]$ ss -nop | grep 1521
tcp   ESTAB  0      0     X.X.0.89:60630    X.X.1.135:1521   timer:(keepalive,14min,0)
[opc@bastion ~]$

So, all good Oracle 26ai Client is supporting Dead Connection Detection with SQLNET.EXPIRE_TIME parameter.

Let’s test it with Oracle 19c Client

We can easily confirm again that Oracle 19c Client does not support Dead Connection Detection on the client side.

Let’s move back to Oracle 19c Client home.

[oracle@bastion admin]$ export PATH=/opt/oracle/client19c/bin

[oracle@bastion admin]$ which sqlplus
/opt/oracle/client19c/bin/sqlplus

Run a sqlplus connection.

[oracle@bastion admin]$ sqlplus test01/test_expire@svl-oat:1521/testz_tmr_003i.db.jewlab.oraclevcn.com

SQL*Plus: Release 19.0.0.0.0 - Production on Thu May 21 16:48:09 2026
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Last Successful login time: Thu May 21 2026 16:46:07 +02:00

Connected to:
Oracle Database 19c EE High Perf Release 19.0.0.0.0 - Production
Version 19.23.0.0.0

SQL>

And check connection configuration.

[opc@bastion ~]$ ss -nop | grep 1521
tcp   ESTAB      0      0    X.X.0.89:64078    X.X.1.135:1521
[opc@bastion ~]$

There is no timer/keepalive handled with Oracle 19c Client.

To wrap up…

Oracle Database 26ai Client is now supporting Dead Connection Detection on the client side. For our customer configuration this will help the client to check every X minutes (EXPIRE_TIME configured value) for Dead Connection. So if the CMAN and listener connections have already died and if for any reason the Network Load Balancer is still keeping the connection with the client, the client will close the connection after X minutes.

L’article Oracle Database 26ai Client and SQLNET.EXPIRE_TIME est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/oracle-database-26ai-client-and-sqlnet-expire_time/feed/ 0 Loosing java remote console full control during ODA Reimage https://www.dbi-services.com/blog/loosing-java-remote-console-full-control-during-oda-reimage/ https://www.dbi-services.com/blog/loosing-java-remote-console-full-control-during-oda-reimage/#respond Wed, 22 Apr 2026 08:18:39 +0000 https://www.dbi-services.com/blog/?p=43970 Recently I had to patch an ODA at one of our customer from 19.20 to 19.26, going through 19.24. As you are aware of, knowing we go from Oracle Linux 7 (19.20) to Oracle Linux 8 (19.24), we need to patch the ODA using Data Preserving Reprovisioning. This includes a reimage of the nodes… Data […]

L’article Loosing java remote console full control during ODA Reimage est apparu en premier sur dbi Blog.

]]> Recently I had to patch an ODA at one of our customer from 19.20 to 19.26, going through 19.24. As you are aware of, knowing we go from Oracle Linux 7 (19.20) to Oracle Linux 8 (19.24), we need to patch the ODA using Data Preserving Reprovisioning. This includes a reimage of the nodes…

Data Preserving Reprovisioning will allow you reprovisioning an already deployed Oracle Database Appliance system keeping the storage and the databases on the appliance, so without any modifications on the information stored in the ASM (storage). The information of the source system will be saved in a server data archive files. The appliance will then be reimaged with Oracle Database Appliance release iso image and the saved metadata will be used to directly reprovision the system and bring back all the resources such as databases, DB systems, Oracle ASR, and others.

The reimage can only be done using the java remote console, available with the ILOM. Remember that to attach the iso on the java console and use it during the reimage, you need to add it and connect it as described in the next picture.

It is important to see the following red message to ensure that the connection with the iso is made and that it can be used.

The problem is that after a time, during the reimage process, the java remote console lost full control on the node, having as consequence a lost of the iso connection and a failure in the reimage process.

This could be seen in the console with the message (View Only) displayed in the bar and the Red Cross on the top of the Computer (screen)/mouse image.

Here:

When it should be:

Here the whole picture showing the connection problem.

And we can see the message : “Sorry, keyboard and mouse connection has been lost. \nIf storage was being shared it has been lost also”.

When a correct connection would be the following one, having (Full Control) message displayed in the bar, with no lost of keyboard/mouse and storage (iso) connection.

After several tries it was always the same problem. After a few minutes, lost of the connection…

I then decided to check the ILOM sessions, connecting to the ILOM through ssh connection.

login as: root
Keyboard-interactive authentication prompts from server:
| Password:
End of keyboard-interactive prompts from server

Oracle(R) Integrated Lights Out Manager

Version 5.1.1.23 r151958

Copyright (c) 2023, Oracle and/or its affiliates. All rights reserved.

Warning: HTTPS certificate is set to factory default.

Hostname: ODA01-ilo

-> show /SP/sessions

 /SP/sessions
    Targets:
        55342
        55346 (current)

    Properties:

    Commands:
        cd
        delete
        show

I found surprising to see so high sessions id. I decided to reset the ILOM. This would of course not restart the node.

-> reset /SP
Are you sure you want to reset /SP (y/n)? y
Performing reset on /SP

And connected again to the ILOM through ssh to checked the sessions:

login as: root
Keyboard-interactive authentication prompts from server:
| Password:
End of keyboard-interactive prompts from server

Oracle(R) Integrated Lights Out Manager

Version 5.1.1.23 r151958

Copyright (c) 2023, Oracle and/or its affiliates. All rights reserved.

Warning: HTTPS certificate is set to factory default.

Hostname: ODA01-ilo

-> show /SP/sessions

 /SP/sessions
    Targets:
        1 (current)

    Properties:

    Commands:
        cd
        delete
        show

The sessions id were back to one, and I did not have any lost connection issue any more. I could successfully reimage the ODA to 19.24 version.

L’article Loosing java remote console full control during ODA Reimage est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/loosing-java-remote-console-full-control-during-oda-reimage/feed/ 0 MySQL Metadata Store service not able to start on an ODA https://www.dbi-services.com/blog/mysql-metadata-store-service-not-able-to-start-on-an-oda/ https://www.dbi-services.com/blog/mysql-metadata-store-service-not-able-to-start-on-an-oda/#respond Tue, 31 Mar 2026 20:32:30 +0000 https://www.dbi-services.com/blog/?p=43699 I was recently doing some consulting on some ODA at one of our customers, and I faced an issue with the DCS Agent and the MySQL Metadata Store. In this blog, I will show you the problem and how I could resolved it. Problem description On an Oracle Database Appliance, the DCS agent is the […]

L’article MySQL Metadata Store service not able to start on an ODA est apparu en premier sur dbi Blog.

]]> I was recently doing some consulting on some ODA at one of our customers, and I faced an issue with the DCS Agent and the MySQL Metadata Store. In this blog, I will show you the problem and how I could resolved it.

Problem description

On an Oracle Database Appliance, the DCS agent is the software providing the full automation to manage the appliance. It will have no impact on the well working of the databases but is mandatory to manage the appliance (patching, create dbhome, create database and any appliance resources). The DCS agent will perform various operations on the appliance, such as provisioning, database lifecycle management, backup and recovery, and storage management. The DCS Agent will use a MySQL database to store the metadata. In order for the DCS Agent to work, the MySQL database needs to be up and running.

On the node 0 of an ODA X9-2-HA, where I was preparing the next database patching operation, I wanted to run some cleanup.

[root@ODA-HA_node0 ~]# odacli cleanup-patchrepo -comp DB,GI -v 19.20.0.0.0
DCS-10009:Failed to create new service job report.

For which I could get a DCS failure.

I checked the job and realised that an ODA cleaning job where still in running status for about 3 months…

[root@ODA-HA_node0 ~]# odacli list-jobs

ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------------
80f2bfbc-a02e-47bb-b57c-754d3d6d3d3a     Server Patching                                                             2025-11-12 08:00:14 CET             Success
2e36d676-f175-402d-a7e4-735b4eb965de     Patch pre-checks for [STORAGE]                                              2025-11-12 08:58:19 CET             Success
...
c4611880-fdc8-44fd-90a5-c2dd27d12f7c     Auto purge job data                                                         2025-12-09 03:59:48 CET             Success
ed09f270-5eff-4141-9b7b-51a2e5d30e32     Auto purge job data                                                         2025-12-10 03:59:50 CET             Success
aecd8ca5-44fb-4e2c-af2b-0e42e954381a     Auto purge job data                                                         2025-12-11 03:59:52 CET             Running

[root@ODA-HA_node0 ~]#

Troubleshooting

I restarted the DCS agent.

[root@ODA-HA_node0 ~]# systemctl stop initdcsagent

[root@ODA-HA_node0 ~]# systemctl start initdcsagent

[root@ODA-HA_node0 ~]# systemctl status initdcsagent
● initdcsagent.service - Oracle dcs-agent startup
   Loaded: loaded (/etc/systemd/system/initdcsagent.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2026-03-18 14:35:41 CET; 2s ago
 Main PID: 92038 (sh)
    Tasks: 96 (limit: 79998)
   Memory: 331.7M
   CGroup: /system.slice/initdcsagent.service
           ├─92038 /bin/sh -c . /opt/oracle/dcs/bin/setupJreAgent.sh;LD_LIBRARY_PATH=/opt/oracle/rhp/lib:$JRE_HOME/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib $JAVA -Xms128m -Xmx256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC -XX:ParallelGCThreads=4 -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/opt/oracle/dcs/log/gc-dcs-agent-%t-%p.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFi>
           └─92039 /opt/oracle/dcs/java/1.8.0_441/bin/java -Xms128m -Xmx256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC -XX:ParallelGCThreads=4 -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/opt/oracle/dcs/log/gc-dcs-agent-%t-%p.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Doracle.security.jps.config=/opt/oracle/dcs/agent/jps-config.xml -Dio.nett>

Mar 18 14:35:41 ODA-HA_node0 systemd[1]: Started Oracle dcs-agent startup.
[root@ODA-HA_node0 ~]# odacli list-jobs | tail -n3

Which did not help. In the DCS Agent log I could find some errors with HAMI. Oracle HAMI is the Oracle High Availability Metadata Infrastructure service providing distributed services required by DCS including locking and synchronizing configuration details in the cluster.

[root@ODA-HA_node0 log]# view dcs-agent-infra.log
...
...
...
2026-03-18 14:59:37,776 DEBUG [DcsInfraProvisioningThread] [] c.o.d.i.h.c.CommandExecutor: Going to execute command '/opt/oracle/dcs/hami/bin/hamictl.sh ssh-equiv --ensemble ODA_DCS --hosts ODA-HA_node0-priv,ODA-HA_node1-priv --json'
2026-03-18 14:59:38,014 DEBUG [DcsInfraProvisioningThread] [] c.o.d.i.h.c.CommandExecutor: Got result from execution of '/opt/oracle/dcs/hami/bin/hamictl.sh ssh-equiv --ensemble ODA_DCS --hosts ODA-HA_node0-priv,ODA-HA_node1-priv --json':
 - RC:     1
 - OUTPUT: /opt/oracle/dcs/oda-python3/lib/python3.12/getpass.py:91: GetPassWarning: Can not control echo on the terminal.
Warning: Password input may be echoed.
hamictl_oda_dcs's password: *********
{"operation": "ssh-equiv", "result": "FAILED", "message": "Invalid password"}

2026-03-18 14:59:38,015 ERROR [DcsInfraProvisioningThread] [] c.o.d.a.i.DcsInfraProvisioningThread: Could not provision DCS infrastructure
com.oracle.dcs.infra.hams.ctl.HamiCtl$HamiCtlException: Invalid password

I decided to restart MySQL service.

[root@ODA-HA_node0 ~]# systemctl stop oda-mysql

[root@ODA-HA_node0 ~]# systemctl start oda-mysql
Job for oda-mysql.service failed because the control process exited with error code.
See "systemctl status oda-mysql.service" and "journalctl -xe" for details.

Which failed…

Checking the logs:

[root@ODA-HA_node0 ~]# journalctl -xe
...
-- Unit oda-mysql.service has begun starting up.
Mar 31 09:23:10 ODA-HA_node0 systemd[90052]: oda-mysql.service: Failed to execute command: Permission denied
Mar 31 09:23:10 ODA-HA_node0 systemd[90052]: oda-mysql.service: Failed at step EXEC spawning /opt/oracle/dcs/mysql/bin/mysqld: Permission denied

I could see that it was complaining about permissions.

I checked the permissions on the MySQL binary:

[root@ODA-HA_node0 ~]# ls -ltrh /opt/oracle/dcs/mysql/bin/mysqld
-rwxr-x--- 1 odamysql odamysql 338M Mar 26 16:55 /opt/oracle/dcs/mysql/bin/mysqld
[root@ODA-HA_node0 ~]#

And that one was ok. The linux user starting MySQL, odamysql, was having the right permissions.

I then checked all permissions including the directories:

[root@ODA-HA_node0 log]# ls -ld /opt /opt/oracle /opt/oracle/dcs /opt/oracle/dcs/mysql /opt/oracle/dcs/mysql/bin
drwxrwx---  11 root     oinstall 4096 Nov 21 13:58 /opt
drwxr-xr-x.  9 root     root     4096 Nov 14 16:31 /opt/oracle
drwxr-xr-x. 30 root     root     4096 Nov 14 12:40 /opt/oracle/dcs
drwxr-xr-x. 13 root     root     4096 Nov 14 09:10 /opt/oracle/dcs/mysql
drwxr-x---.  2 odamysql odamysql 4096 Mar 26 16:55 /opt/oracle/dcs/mysql/bin
[root@ODA-HA_node0 log]#

And I could see that there were wrong on the /opt directory… The odamysql directory is not able to enter and read in the /opt directory, as it does not belong to the group and there is no permissions on the other users…

I checked on a valid ODA and as expected the permissions should be:

drwxr-xr-x. 11 root     root     4096 Mar 18 10:02 /opt

So this is mainly due to an human mistake…

Resolution

I first checked and could confirm the issue only affect the /opt directory itself.

[root@ODA-HA_node0 opt]# pwd
/opt

[root@ODA-HA_node0 opt]# ls -ltrh
total 48K
drwxr-xr-x.  3 root root     4.0K Sep 17  2024 lsi
drwxr-xr-x   2 root root     4.0K Jan 15  2025 oracle.ahf
drwx------.  2 root root      16K Nov 11 15:42 lost+found
drwxr-xr-x.  4 root root     4.0K Nov 11 16:32 MegaRAID
drwxr-xr-x   3 root root     4.0K Nov 12 10:20 ORCLfmap
drwxr-xr-x   5 root root     4.0K Nov 12 13:36 odabr
drwxr-xr-x.  9 root root     4.0K Nov 14 16:31 oracle
drwxrwx---  16 root oinstall 4.0K Feb 16 11:19 commvault
drwx------  11 root root     4.0K Mar 30 02:08 splunkforwarder
[root@ODA-HA_node0 opt]#

I change the permissions for other users.

[root@ODA-HA_node0 /]# chmod o+rx /opt

[root@ODA-HA_node0 /]# ls -ld /opt /opt/oracle /opt/oracle/dcs /opt/oracle/dcs/mysql /opt/oracle/dcs/mysql/bin
drwxrwxr-x  11 root     oinstall 4096 Nov 21 13:58 /opt
drwxr-xr-x.  9 root     root     4096 Nov 14 16:31 /opt/oracle
drwxr-xr-x. 30 root     root     4096 Nov 14 12:40 /opt/oracle/dcs
drwxr-xr-x. 13 root     root     4096 Nov 14 09:10 /opt/oracle/dcs/mysql
drwxr-x---.  2 odamysql odamysql 4096 Mar 26 16:55 /opt/oracle/dcs/mysql/bin
[root@ODA-HA_node0 /]#

And I could successfully start the MySQL service.

[root@ODA-HA_node0 /]# systemctl start oda-mysql

[root@ODA-HA_node0 /]# systemctl status oda-mysql
● oda-mysql.service - MySQL Server for ODA
   Loaded: loaded (/etc/systemd/system/oda-mysql.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2026-03-31 09:55:48 CEST; 6s ago
     Docs: man:mysqld(8)
           http://dev.mysql.com/doc/refman/en/using-systemd.html
 Main PID: 55837 (mysqld)
   Status: "Server is operational"
    Tasks: 33 (limit: 79998)
   Memory: 423.5M
   CGroup: /system.slice/oda-mysql.service
           └─55837 /opt/oracle/dcs/mysql/bin/mysqld --defaults-file=/opt/oracle/dcs/mysql/etc/mysqldb.cnf --skip-mysqlx

Mar 31 09:55:48 ODA-HA_node0 systemd[1]: Starting MySQL Server for ODA...
Mar 31 09:55:48 ODA-HA_node0 systemd[1]: Started MySQL Server for ODA.
[root@ODA-HA_node0 /]#

And I could successfully start the DCS Agent.

[root@ODA-HA_node0 /]# systemctl status initdcsagent
● initdcsagent.service - Oracle dcs-agent startup
   Loaded: loaded (/etc/systemd/system/initdcsagent.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Wed 2026-03-18 15:22:48 CET; 1 weeks 5 days ago
 Main PID: 84546 (code=killed, signal=TERM)

Mar 19 16:18:04 ODA-HA_node0 systemd[1]: Dependency failed for Oracle dcs-agent startup.
Mar 19 16:18:04 ODA-HA_node0 systemd[1]: initdcsagent.service: Job initdcsagent.service/start failed with result 'dependency'.
Mar 24 13:38:38 ODA-HA_node0 systemd[1]: Dependency failed for Oracle dcs-agent startup.
Mar 24 13:38:38 ODA-HA_node0 systemd[1]: initdcsagent.service: Job initdcsagent.service/start failed with result 'dependency'.
Mar 25 09:46:35 ODA-HA_node0 systemd[1]: Dependency failed for Oracle dcs-agent startup.
Mar 25 09:46:35 ODA-HA_node0 systemd[1]: initdcsagent.service: Job initdcsagent.service/start failed with result 'dependency'.
Mar 26 11:06:50 ODA-HA_node0 systemd[1]: Dependency failed for Oracle dcs-agent startup.
Mar 26 11:06:50 ODA-HA_node0 systemd[1]: initdcsagent.service: Job initdcsagent.service/start failed with result 'dependency'.
Mar 30 16:07:15 ODA-HA_node0 systemd[1]: Dependency failed for Oracle dcs-agent startup.
Mar 30 16:07:15 ODA-HA_node0 systemd[1]: initdcsagent.service: Job initdcsagent.service/start failed with result 'dependency'.

[root@ODA-HA_node0 /]# systemctl start initdcsagent

[root@ODA-HA_node0 /]# systemctl status initdcsagent
● initdcsagent.service - Oracle dcs-agent startup
   Loaded: loaded (/etc/systemd/system/initdcsagent.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2026-03-31 09:57:34 CEST; 1s ago
 Main PID: 59253 (sh)
    Tasks: 96 (limit: 79998)
   Memory: 270.0M
   CGroup: /system.slice/initdcsagent.service
           ├─59253 /bin/sh -c . /opt/oracle/dcs/bin/setupJreAgent.sh;LD_LIBRARY_PATH=/opt/oracle/rhp/lib:$JRE_HOME/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib $JAVA -Xms128m -Xmx256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC -XX:ParallelGCThreads=4 -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/opt/oracle/dcs/log/gc-dcs-agent-%t-%p.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFi>
           └─59254 /opt/oracle/dcs/java/1.8.0_441/bin/java -Xms128m -Xmx256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC -XX:ParallelGCThreads=4 -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/opt/oracle/dcs/log/gc-dcs-agent-%t-%p.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Doracle.security.jps.config=/opt/oracle/dcs/agent/jps-config.xml -Dio.nett>

Mar 31 09:57:34 ODA-HA_node0 systemd[1]: Started Oracle dcs-agent startup.
[root@ODA-HA_node0 /]#

And I could successfully run the cleanup:

[root@ODA-HA_node0 ~]# odacli cleanup-patchrepo -comp DB,GI -v 19.20.0.0.0
{
  "jobId" : "08b395be-ae5a-4d6f-8e5d-1c0fa232f672",
  "status" : "Created",
  "message" : "",
  "reports" : [ ],
  "createTimestamp" : "March 31, 2026 11:21:58 CEST",
  "resourceList" : [ ],
  "description" : "Cleanup patchrepos",
  "updatedTime" : "March 31, 2026 11:21:58 CEST",
  "jobType" : null,
  "cpsMetadata" : null
}

[root@ODA-HA_node0 ~]# odacli describe-job -i 08b395be-ae5a-4d6f-8e5d-1c0fa232f672

Job details
----------------------------------------------------------------
                     ID:  08b395be-ae5a-4d6f-8e5d-1c0fa232f672
            Description:  Cleanup patchrepos
                 Status:  Success
                Created:  March 31, 2026 11:21:58 CEST
                Message:

Task Name                                Node Name                 Start Time                               End Time                                 Status
---------------------------------------- ------------------------- ---------------------------------------- ---------------------------------------- ----------------
Cleanup Repository                       ODA-HA_node1                 March 31, 2026 11:21:58 CEST             March 31, 2026 11:21:59 CEST             Success
Cleanup old ASR rpm                      ODA-HA_node0                 March 31, 2026 11:21:59 CEST             March 31, 2026 11:21:59 CEST             Success

[root@ODA-HA_node0 ~]#

Finally, I have also changed the group for /opt accordingly to root group.

[root@ODA-HA_node0 /]# chgrp root /opt

[root@ODA-HA_node0 /]# ls -ld /opt /opt/oracle /opt/oracle/dcs /opt/oracle/dcs/mysql /opt/oracle/dcs/mysql/bin
drwxrwxr-x  11 root     root     4096 Nov 21 13:58 /opt
drwxr-xr-x.  9 root     root     4096 Nov 14 16:31 /opt/oracle
drwxr-xr-x. 30 root     root     4096 Nov 14 12:40 /opt/oracle/dcs
drwxr-xr-x. 13 root     root     4096 Nov 14 09:10 /opt/oracle/dcs/mysql
drwxr-x---.  2 odamysql odamysql 4096 Mar 26 16:55 /opt/oracle/dcs/mysql/bin
[root@ODA-HA_node0 /]#

To wrap up…

DCS Agent and MySQL database will have no effect on the well working of your databases, but if they are not working you are not able to manage your appliance any more. Here, wrong permissions (for sure a human mistake) had sad effects on appliance software. Solving the permissions issue resolved my problem.

L’article MySQL Metadata Store service not able to start on an ODA est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/mysql-metadata-store-service-not-able-to-start-on-an-oda/feed/ 0 Starting ODA 19.29 – Old grid version cleanup https://www.dbi-services.com/blog/starting-oda-19-29-old-grid-version-cleanup/ https://www.dbi-services.com/blog/starting-oda-19-29-old-grid-version-cleanup/#respond Mon, 30 Mar 2026 17:05:27 +0000 https://www.dbi-services.com/blog/?p=43683 We know that during the ODA patching, a new grid version will be installed. To avoid the /u01 file system to fill up, we need to cleanup the old version. In this blog, I will show you how you can do it and also few changes coming with version 19.29, as you might need a […]

L’article Starting ODA 19.29 – Old grid version cleanup est apparu en premier sur dbi Blog.

]]> We know that during the ODA patching, a new grid version will be installed. To avoid the /u01 file system to fill up, we need to cleanup the old version. In this blog, I will show you how you can do it and also few changes coming with version 19.29, as you might need a reboot before.

Remember that until 19.28, the patch of the server was done with odacli update-server command, and this will patch the whole server (OS, ILOM, BIOS, …) and the grid in the same step.

Starting 19.29 you will run the server and the grid patching separately. odacli update-servercomponents will patch the server (OS, ILOM, HMP, firmware for local disk and controller, …) and odacli update-gihome will patch and install the new grid version. A reboot of the node will only be performed after the servercomponents update due to the ILOM and BIOS update.

Once the patching process was successfully done, we can cleanup old grid version. In my example, I patched the ODA from version 19.28 to 19.30.

Check usage

First of all we check usage of the grid files. The new grid version is 19.30, so I expect to have file using 19.30 grid.

[root@ODA01 ~]# lsof | grep "/u01/app/19.30.0.0/" | wc -l
29545

Before any cleanup, I need to ensure the old 19.28 grid infrastructure is not used any more.

[root@ODA01 ~]# lsof | grep "/u01/app/19.28.0.0/" | wc -l
3295

Which is definitely note the case.

I check which process are still using old grid infra.

[root@ODA01 ~]# lsof | grep "/u01/app/19.28.0.0/" | grep -i oakd | wc -l
3233

[root@ODA01 ~]# lsof | grep "/u01/app/19.28.0.0/" | grep -iv oakd | wc -l
62

[root@ODA01 ~]# lsof | grep "/u01/app/19.28.0.0/" | grep -iv oakd
java      86061                            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061  2390 LegacyRun            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061  2394 LegacyRun            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061  2440 LegacyRun            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061  2545 LegacyRun            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061  2546 LegacyRun            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061  2547 LegacyRun            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061  2548 LegacyRun            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061  4274 ForkJoinP            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 16159 process              root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86062 TFAServer            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86063 ParGC\x20            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86064 ParGC\x20            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86065 ParGC\x20            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86066 VM\x20Thr            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86068 Reference            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86069 Finalizer            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86071 Signal               root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86072 Service              root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86073 C2\x20Com            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86074 C1\x20Com            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86075 Sweeper              root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86078 VM\x20Per            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86079 Common-Cl            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86145 process              root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86252 LogFlushe            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86253 DataCorru            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86296 INCompres            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86297 Cleaner-1            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86298 Checkpoin            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86396 pool-3-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86397 pool-3-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86398 pool-3-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86551 TFAIndexe            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86983 TFAMonito            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87763 ActionHan            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87776 ClusterEv            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87777 Repositor            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87782 PeriodicC            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87786 InstanceM            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87788 Thread-13            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87790 DiskUsage            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87794 process              root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87797 pool-2-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87799 PurgeMana            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87800 TFAMaster            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87804 LegacyTel            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87805 TFAPlugin            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87806 pool-2-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87807 TFAInvent            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87808 TFADbUtlS            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87809 TFADbUtlP            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87810 RetryUplo            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 88715 Tailing              root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 88718 Tailing              root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 88719 Tailing              root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 88720 Tailing              root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 91944 pool-6-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 98253 pool-2-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 98254 pool-2-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 98331 pool-2-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 98388 pool-2-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar

[root@ODA01 ~]# lsof | grep "/u01/app/19.28.0.0/" | grep -i oakd | tail -n3
oakd      65702 71035 oakd                 root  118r      REG             252,10       174592    1717769 /u01/app/19.28.0.0/grid/rdbms/mesg/diaus.msb
oakd      65702 71035 oakd                 root  119r      REG             252,10       174592    1717769 /u01/app/19.28.0.0/grid/rdbms/mesg/diaus.msb
oakd      65702 71035 oakd                 root  120r      REG             252,10       174592    1717769 /u01/app/19.28.0.0/grid/rdbms/mesg/diaus.msb
[root@ODA01 ~]#

So OAK and TFA are still running on old grid version and using old grid jar library.

This comes from the fact that after the server components patches, the server is rebooted, the oak, tfa and other process will start with the existing grid version, 19.28. The grid patch will not reboot the server and not restart those processes, so they will still run with old grid infra.

This is why before any cleanup and starting 19.29 only, we need to reboot the ODA.

Reboot the DOA:

[root@ODA01 ~]# systemctl reboot
[root@ODA01 ~]#

After reboot we can see that only new grid version is used:

[root@ODA01 ~]# uptime
 16:16:17 up 3 min,  1 user,  load average: 2.06, 1.58, 0.65

[root@ODA01 ~]# lsof | grep "/u01/app/19.28.0.0/" | wc -l
0

[root@ODA01 ~]# lsof | grep "/u01/app/19.30.0.0" | wc -l
45798
[root@ODA01 ~]#

Cleanup old grid

Current space for /u1 is 6.2 GB:

[root@ODA01 ~]# df -h / /opt /u01
Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/VolGroupSys-LogVolRoot   30G  5.1G   23G  19% /
/dev/mapper/VolGroupSys-LogVolOpt    48G   26G   20G  57% /opt
/dev/mapper/VolGroupSys-LogVolU01    49G   41G  6.2G  87% /u01
[root@ODA01 ~]#

Remove old version of grid:

[root@ODA01 ~]# cd /u01/app

[root@ODA01 app]# mv 19.28.0.0 BAK_19.28.0.0

[root@ODA01 app]# lsof | grep "/u01/app/19.28.0.0" | wc -l
0

[root@ODA01 app]# lsof | grep "/u01/app/BAK_19.28.0.0" | wc -l
0

[root@ODA01 app]# rm -rf BAK_19.28.0.0
[root@ODA01 app]#

Check current space:

[root@ODA01 app]# df -h / /u01 /opt /boot
Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/VolGroupSys-LogVolRoot   30G  5.1G   23G  19% /
/dev/mapper/VolGroupSys-LogVolU01    49G   27G   20G  58% /u01
/dev/mapper/VolGroupSys-LogVolOpt    48G   26G   20G  57% /opt
/dev/md126p2                        488M  218M  235M  49% /boot
[root@ODA01 app]#

We have taken back about 14 GB.

Check oracle inventory

Checking oracle inventory:

[root@ODA01 app]# cd /u01/app/oraInventory/ContentsXML/

[root@ODA01 ContentsXML]# cat inventory.xml

You will see both grid home:

<HOME NAME="OraGrid192800" LOC="/u01/app/19.28.0.0/grid" TYPE="O" IDX="13"/>
<HOME NAME="OraGrid193000" LOC="/u01/app/19.30.0.0/grid" TYPE="O" IDX="15" CRS="true"/>

The current running one is the one having CRS option to true.

Detach old grid home from the inventory

We can now detach old grid from the inventory:

[grid@ODA01 ~]$ echo $ORACLE_SID
+ASM1

[grid@ODA01 ~]$ echo $ORACLE_HOME
/u01/app/19.30.0.0/grid

[grid@ODA01 ~]$ cd $ORACLE_HOME/oui/bin

[grid@ODA01 bin]$ ./runInstaller -detachHome -silent -local ORACLE_HOME=/u01/app/19.28.0.0/grid
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 24575 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
You can find the log of this install session at:
 /u01/app/oraInventory/logs/DetachHome2026-03-29_04-34-43PM.log
'DetachHome' was successful.

Check inventory

Checking the inventory again:

[grid@ODA01 bin]$ cd /u01/app/oraInventory/ContentsXML/

[grid@ODA01 ContentsXML]$ cat inventory.xml

We will see old grid with option REMOVE=”T”

<HOME NAME="OraGrid193000" LOC="/u01/app/19.30.0.0/grid" TYPE="O" IDX="15" CRS="true"/>
<HOME NAME="OraGrid192800" LOC="/u01/app/19.28.0.0/grid" TYPE="O" IDX="13" REMOVED="T"/>

L’article Starting ODA 19.29 – Old grid version cleanup est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/starting-oda-19-29-old-grid-version-cleanup/feed/ 0 ODA 19.30 prepatch report failing on OS patches https://www.dbi-services.com/blog/oda-19-30-prepatch-report-failing-on-os-patches/ https://www.dbi-services.com/blog/oda-19-30-prepatch-report-failing-on-os-patches/#respond Mon, 30 Mar 2026 15:03:50 +0000 https://www.dbi-services.com/blog/?p=43662 I had to recently patch several ODAs from 19.28 to 19.30, and one ODA patching failed running the prepatch report which is mandatory and helpful before going to patch the ODA. For each step (server, grid or storage) a prepatch report is needed. I had to troubleshoot and I could solve the problem, and I […]

L’article ODA 19.30 prepatch report failing on OS patches est apparu en premier sur dbi Blog.

]]> I had to recently patch several ODAs from 19.28 to 19.30, and one ODA patching failed running the prepatch report which is mandatory and helpful before going to patch the ODA. For each step (server, grid or storage) a prepatch report is needed. I had to troubleshoot and I could solve the problem, and I thought it would be more than interesting to share it, expecting it might be helpful.

Problem description

After dcs-admin update, dcs components update, repository update with grid and rdbms clones, I have run a prepatch report, and decided to run it already for all components : server, grid and storage.

This is why we will see in the output following part : OS, ILOM, ORACHKSERVER, SERVER, STORAGE, RHPGI, GI.

Running the prepatch report:

[root@ODA01 ~]# /opt/oracle/dcs/bin/odacli create-prepatchreport -sc -gi -st -v 19.30.0.0.0

Job details
----------------------------------------------------------------
                     ID:  6681af21-907a-4ffc-8add-d40af11441ae
            Description:  Patch pre-checks for [OS, ILOM, ORACHKSERVER, SERVER, STORAGE, RHPGI, GI] to 19.30.0.0.0
                 Status:  Created
                Created:  March 29, 2026 10:35:01 CEST
                Message:  Use 'odacli describe-prepatchreport -i 6681af21-907a-4ffc-8add-d40af11441ae' to check details of results

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------

[root@ODA01 ~]#

Describing the report, I could see that it failed on the OS rpm verification:

[root@ODA01 ~]# odacli describe-prepatchreport -i 6681af21-907a-4ffc-8add-d40af11441ae

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  6681af21-907a-4ffc-8add-d40af11441ae
            Description:  Patch pre-checks for [OS, ILOM, ORACHKSERVER, SERVER, STORAGE, RHPGI, GI] to 19.30.0.0.0
                 Status:  FAILED
                Created:  March 29, 2026 10:35:01 AM CEST
                 Result:  One or more pre-checks failed for [OS]

Node Name
---------------
ODA01

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is patch location available     Success   Patch location is available.
Verify All OS patches           Failed    DCS-10252 - dnf dry-run execution
                                          failed: check
                                          /opt/oracle/dcs/log/jobfiles/
                                          dnfdryrunout_2026-03-29_10-35-
                                          15.0673_536.log file for more details
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate all KVM ACFS           Success   All KVM ACFS resources are running
resources are running
Validate DB System VM states    Success   All DB System VMs states are expected
Validate command execution      Success   Validated command execution

__STORAGE__
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Patch location validation       Success   Verified patch location
Patch tag validation            Success   Verified patch tag
Storage patch tag validation    Success   Verified storage patch location
Verify ASM disks status         Success   ASM disks are online
Validate command execution      Success   Validated command execution

__RHPGI__
Validate available space        Success   Validated free space under /u01
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Successfully validated minimum version
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM is online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Validate space for clones       Success   Clones volume is already created
volume
Validate command execution      Success   Validated command execution

[root@ODA01 ~]#

I checked the appropriate log, and could easily guess that the process complained about missing rhn-client-tools:

[root@ODA01 ~]# view /opt/oracle/dcs/log/jobfiles/dnfdryrunout_2026-03-29_10-35-15.0673_536.log
2026-03-29 10:35:29,977 [DNFHelper] INFO    : File path /tmp/dcsfiles/dnfoperation_2026-03-29_10-35-15.0676_536.json exists
2026-03-29 10:35:29,977 [DNFHelper] INFO    : File path /tmp/dcsfiles/dnfoperation_2026-03-29_10-35-15.0676_536.json is a regular file
2026-03-29 10:35:29,979 [dnf      ] DEBUG   : User-Agent: constructed: 'libdnf (Oracle Linux Server 8.10; server; Linux.x86_64)'
2026-03-29 10:35:29,991 [dnf      ] DEBUG   : User-Agent: constructed: 'libdnf (Oracle Linux Server 8.10; server; Linux.x86_64)'
2026-03-29 10:35:30,139 [dnf      ] DEBUG   : repo: using cache for: PrecheckOSPatchRepo
2026-03-29 10:35:30,147 [dnf      ] DEBUG   : PrecheckOSPatchRepo: using metadata from Sun 29 Mar 2026 10:35:12 AM CEST.
2026-03-29 10:35:30,148 [dnf      ] INFO    : Last metadata expiration check: 0:00:18 ago on Sun 29 Mar 2026 10:35:12 AM CEST.
2026-03-29 10:35:31,907 [DNFHelper] DEBUG   : === DNF Plugin Status ===
2026-03-29 10:35:31,907 [DNFHelper] DEBUG   : DNF plugins are enabled
2026-03-29 10:35:31,910 [DNFHelper] DEBUG   : === Debug Info ===
2026-03-29 10:35:31,910 [DNFHelper] DEBUG   : Operation RPM Metadata: {"install": ["hmpipmitool.x86_64", "rhn-setup.x86_64", "uptrack-updates-5.15.0-316.196.4.1.el8uek.x86_64.noarch", "ksplice-offline.x86_64", "ksplice.x86_64"], "exclude": ["linux-firmware-core.noarch"], "remove": ["uptrack-updates-5.15.0-310.184.5.2.el8uek.x86_64.noarch"]}
2026-03-29 10:35:31,910 [DNFHelper] DEBUG   : Installed packages: ['MegaCli-8.07.07-1.noarch', 'NetworkManager-1:1.40.16-19.0.1.el8_10.x86_64', 'NetworkManager-config-server-1:1.40.16-19.0.1.el8_10.noarch', 'NetworkManager-libnm-1:1.40.16-19.0.1.el8_10.x86_64', 'NetworkManager-team-1:1.40.16-19.0.1.el8_10.x86_64', 'NetworkManager-tui-1:1.40.16-19.0.1.el8_10.x86_64', 'VRTSnbcfg-10.2.0.1-1686143553.x86_64', 'VRTSnbclibs-10.2.0.1-1686143611.x86_64', 'VRTSnbclt-10.2.0.1-1686143729.x86_64', 'VRTSnbpck-10.2.0.1-1686143786.x86_64', 'VRTSpbx-1.17.62.5-5.x86_64', 'VRTSpddea-18.0.1.0-0023.x86_64', 'abattis-cantarell-fonts-0.0.25-6.el8.noarch', 'acl-2.2.53-3.el8.x86_64', 'adcli-0.9.2-1.el8.x86_64', 'adwaita-cursor-theme-3.28.0-3.el8.noarch', 'adwaita-icon-theme-3.28.0-3.el8.noarch', 'aide-0.16-14.el8_5.1.x86_64', 'alsa-lib-1.2.10-2.el8.x86_64', 'annobin-11.13-2.0.6.el8.x86_64', 'at-3.1.20-12.el8.x86_64', 'at-spi2-atk-2.26.2-1.el8.x86_64', 'at-spi2-core-2.28.0-1.el8.x86_64', 'atk-2.28.1-1.el8.x86_64', 'attr-2.4.48-3.el8.x86_64', 'audit-3.1.2-1.0.1.el8.x86_64', 'audit-libs-3.1.2-1.0.1.el8.x
...
...
...
2026-03-29 10:35:32,505 [DNFHelper] ERROR   : Dependency solving failed
2026-03-29 10:35:32,505 [DNFHelper] ERROR   : Error occured:
Traceback (most recent call last):
  File "/opt/oracle/oak/pkgrepos/scripts/19.30.0.0.0/dnfhelper.py", line 315, in main
    metadata_file=metadata_file
  File "/opt/oracle/oak/pkgrepos/scripts/19.30.0.0.0/dnfhelper.py", line 226, in run_combined_dnf_txn
    base.resolve()
  File "/usr/lib/python3.6/site-packages/dnf/base.py", line 906, in resolve
    raise exc
dnf.exceptions.DepsolveError:
 Problem: conflicting requests
  - nothing provides rhn-client-tools = 2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe needed by rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64 from PrecheckOSPatchRepo

Analysing problem and root cause

I first checked if the package is really missing:

[root@ODA01 ~]# rpm -qa | grep -i rhn-client-tools
[root@ODA01 ~]#

And it was definitively missing, when on other ODA it was existing:

[root@ODA02 ~]# rpm -qa | grep -i rhn-client-tools
python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
[root@ODA02 ~]#

I checked in the history of the package management if one transaction could have affected this.

[root@ODA01 ~]# dnf history | grep -i rhn-client-tools
     2 | remove -y rhn-client-tools                                    | 2024-03-12 17:27 | Removed        |   16  <
[root@ODA01 ~]#

And it was the case! Someone manually removed the package 2 years ago. And also we can see that 16 dependencies rpm have been removed as well. What a mess!

I take the opportunity here to remind that an ODA IS AN APPLIANCE, and not a normal linux server. Any yum or dnf changes will have huge impact on the ODA patching, knowing that each version is coming with its bundle patch version, and the ODA will lock the repository. If a package is needed (for example, veeam agent plugin) it is always better to prioritise an installation with a tar file which will be totally transparent for the ODA patching. Otherwise it is better to install each rpm individually, avoiding if possible yum or dnf. The worst of course is removing a package with several dependencies. dnf remove including dependencies is of course something that I would never recommend to run on an ODA.

The second question we can raise, is why now? We have been running several version from 2024 to 2026 and it seems only the 19.30 version is complaining about it.

Let’s list all the dependant packages that have been removed during this dnf transaction…

[root@ODA01 ~]# dnf history info 2
Transaction ID : 2
Begin time     : Tue 12 Mar 2024 05:27:35 PM CET
Begin rpmdb    : 1162:21b3b42bf2d4360e1ace842a41737674018ff3da
End time       : Tue 12 Mar 2024 05:27:36 PM CET (1 seconds)
End rpmdb      : 1146:a140a149ea28fdac865a8abdbe954771f896a078
User           : root 
Return-Code    : Success
Releasever     : 8
Command Line   : remove -y rhn-client-tools
Comment        :
Packages Altered:
    Removed dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch         @@System
    Removed python3-dmidecode-3.12.3-2.el8.x86_64                                          @@System
    Removed python3-dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch @@System
    Removed python3-dnf-plugin-ulninfo-0.3-2.module+el8.4.0+20142+f119a2fa.noarch          @@System
    Removed python3-hwdata-2.3.6-3.el8.noarch                                              @@System
    Removed python3-librepo-1.14.2-4.el8.x86_64                                            @@System
    Removed python3-netifaces-0.10.6-4.el8.x86_64                                          @@System
    Removed python3-newt-0.52.20-11.el8.x86_64                                             @@System
    Removed python3-rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64           @@System
    Removed python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64    @@System
    Removed python3-rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64           @@System
    Removed rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                   @@System
    Removed rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64            @@System
    Removed rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                   @@System
    Removed rhnsd-5.0.35-3.0.1.module+el8+5192+3173336a.x86_64                             @@System
    Removed usermode-1.113-2.el8.x86_64                                                    @@System
[root@ODA01 yum.repos.d]#

Of course, I do not need to mention that such action has not been performed on the other ODA from the customer field, and this is why the patching did not raise this issue on the other ODAs.

[root@ODA02 ~]# dnf history | grep -i rhn-client-tools
[root@ODA02 ~]#

Resolution

The ODA had no internet access. That was a pity as I could have easily installed the needed packages that were removed from the official oracle linux distribution package manager, using dnf. So the only way of resolution for me was to download and install each package manually. In any case it was not worth to try otherwise the patching activity could have been stopped here. Stuff work as I had to download and install about 16 RPMs.

At the beginning, I tried to find the needed rpm from the Oracle linux distribution (yum.oracle.com) directly, but I need to say that it was not that so easy to find all the rpm. Also the maintenance windows was running on, and I had to find quickly a solution. I then get used of pkgs.org project which is a kind of search engine linking to the appropriate distribution mirror to download the package. We can see the home page of each project. It was much helpful because you can easily find the package, and in the download part it refers you to the exact rpm package you need from the official documentation. I found it so helpful knowing the time constraints.

For example for rhn-setup package, we can easily find it from pkgs.org:

https://oraclelinux.pkgs.org/8/ol8-appstream-x86_64/rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm.html

In the download part you will find the exact distribution link for the source and the binary, as well as the mirror, as displayed in next table:

TypeURL
Binary Packagehttps://yum.oracle.com/repo/OracleLinux/OL8/appstream/x86_64/getPackage/rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm
Source Packagehttps://yum.oracle.com/repo/OracleLinux/OL8/appstream/x86_64/getPackageSource/rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.src.rpm
Mirroryum.oracle.com

So the package comes from the official distribution, and you can confirm for each of your downloads.

I could install the first package which are needed for rhn-client-tools:

[root@ODA01 19.30]# rpm -ivh python3-newt-0.52.20-11.el8.x86_64.rpm
warning: python3-newt-0.52.20-11.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ad986da3: NOKEY
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:python3-newt-0.52.20-11.el8      ################################# [100%]

[root@ODA01 19.30]# rpm -ivh python3-netifaces-0.10.6-4.el8.x86_64.rpm
warning: python3-netifaces-0.10.6-4.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ad986da3: NOKEY
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:python3-netifaces-0.10.6-4.el8   ################################# [100%]

[root@ODA01 19.30]# rpm -ivh python3-hwdata-2.3.6-3.el8.noarch.rpm
warning: python3-hwdata-2.3.6-3.el8.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID ad986da3: NOKEY
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:python3-hwdata-2.3.6-3.el8       ################################# [100%]

[root@ODA01 19.30]# rpm -ivh python3-dmidecode-3.12.2-15.el8.x86_64.rpm
warning: python3-dmidecode-3.12.2-15.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ad986da3: NOKEY
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:python3-dmidecode-3.12.2-15.el8  ################################# [100%]

I then tried to install rhn-client-tools and python3-rhn-client-tools:

[root@ODA01 19.30]# rpm -ivh python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm
warning: python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ad986da3: NOKEY
error: Failed dependencies:
        rhn-client-tools = 2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe is needed by python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64

[root@ODA01 19.30]# rpm -ivh rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm
warning: rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ad986da3: NOKEY
error: Failed dependencies:
        python3-rhn-client-tools = 2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe is needed by rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
[root@ODA01 19.30]#

But it was like each one needs the other, so it was not possible to install it with rpm binary. I then used dnf to install them.

[root@ODA01 19.30]# mkdir local_repo_issue_rhn
[root@ODA01 19.30]# mv rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm ./local_repo_issue_rhn/
[root@ODA01 19.30]# mv python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm ./local_repo_issue_rhn/
[root@ODA01 19.30]# cd local_repo_issue_rhn/
[root@ODA01 local_repo_issue_rhn]# dnf install *.rpm
BKB_Common                                                                                                                                                                                                                            95 kB/s | 1.5 kB     00:00
Dependencies resolved.
=====================================================================================================================================================================================================================================================================
 Package                                                           Architecture                                    Version                                                                               Repository                                             Size
=====================================================================================================================================================================================================================================================================
Installing:
 python3-rhn-client-tools                                          x86_64                                          2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe                                           @commandline                                          110 k
 rhn-client-tools                                                  x86_64                                          2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe                                           @commandline                                          394 k

Transaction Summary
=====================================================================================================================================================================================================================================================================
Install  2 Packages

Total size: 503 k
Installed size: 2.4 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                             1/1
  Installing       : rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                         1/2
  Running scriptlet: rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                         1/2
  Installing       : python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                 2/2
  Running scriptlet: python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                 2/2
  Verifying        : python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                 1/2
  Verifying        : rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                         2/2

Installed:
  python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                           rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64

Complete!
[root@ODA01 local_repo_issue_rhn]#

Working, I had now the rhn-client-tools installed:

[root@ODA01 local_repo_issue_rhn]# rpm -qa | grep -i rhn-client
rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
[root@ODA01 local_repo_issue_rhn]#

I tried a new prepatch report:

[root@ODA01 local_repo_issue_rhn]# cd
[root@ODA01 ~]# /opt/oracle/dcs/bin/odacli create-prepatchreport -sc -gi -st -v 19.30.0.0.0

Job details
----------------------------------------------------------------
                     ID:  0af9bc46-2bfe-42a1-a6dc-bb89b9c0abb8
            Description:  Patch pre-checks for [OS, ILOM, ORACHKSERVER, SERVER, STORAGE, RHPGI, GI] to 19.30.0.0.0
                 Status:  Created
                Created:  March 29, 2026 12:57:10 CEST
                Message:  Use 'odacli describe-prepatchreport -i 0af9bc46-2bfe-42a1-a6dc-bb89b9c0abb8' to check details of results

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------

[root@ODA01 ~]#

Which failed again on the same OS patches verification step:

[root@ODA01 ~]# odacli describe-prepatchreport -i 0af9bc46-2bfe-42a1-a6dc-bb89b9c0abb8

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  0af9bc46-2bfe-42a1-a6dc-bb89b9c0abb8
            Description:  Patch pre-checks for [OS, ILOM, ORACHKSERVER, SERVER, STORAGE, RHPGI, GI] to 19.30.0.0.0
                 Status:  FAILED
                Created:  March 29, 2026 12:57:10 PM CEST
                 Result:  One or more pre-checks failed for [OS]

Node Name
---------------
ODA01

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is patch location available     Success   Patch location is available.
Verify All OS patches           Failed    DCS-10252 - dnf dry-run execution
                                          failed: check
                                          /opt/oracle/dcs/log/jobfiles/
                                          dnfdryrunout_2026-03-29_12-57-
                                          23.0066_1614.log file for more details
...
...
...

And the log showed other missing packages that was part of the dependencies removed by the manual dnf remove command.

[root@ODA01 ~]# view /opt/oracle/dcs/log/jobfiles/dnfdryrunout_2026-03-29_12-57-23.0066_1614.log
...
...
...
dnf.exceptions.DepsolveError:
 Problem: package rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64 from PrecheckOSPatchRepo requires rhnsd, but none of the providers can be installed
  - package rhnsd-5.0.35-3.0.2.module+el8.10.0+90373+b70ceaf0.x86_64 from PrecheckOSPatchRepo requires rhn-check >= 0.0.8, but none of the providers can be installed
  - conflicting requests
  - nothing provides dnf-plugin-spacewalk >= 2.4.0 needed by rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64 from PrecheckOSPatchRepo

So I installed all other dependencies rpm.

I installed usermode rpm:

[root@ODA01 19.30]# rpm -ivh usermode-1.113-2.el8.x86_64.rpm
warning: usermode-1.113-2.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ad986da3: NOKEY
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing..a
   1:usermode-1.113-2.el8             ################################# [100%]
[root@ODA01 19.30]#

And finally all others with dnf:

[root@ODA01 local_repo_issue_rhn]# ls -ltrh
total 568K
-rw-r--r--. 1 root root 33K Mar 29 13:24 rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm
-rw-r--r--. 1 root root 26K Mar 29 13:31 rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm
-rw-r--r--. 1 root root 72K Mar 29 13:31 python3-rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm
-rw-r--r--. 1 root root 52K Mar 29 13:41 rhnsd-5.0.35-3.0.1.module+el8+5192+3173336a.x86_64.rpm
-rw-r--r--. 1 root root 31K Mar 29 13:53 python3-dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch.rpm
-rw-r--r--. 1 root root 54K Mar 29 14:08 python3-librepo-1.14.2-4.el8.x86_64.rpm
-rw-r--r--. 1 root root 24K Mar 29 14:08 dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch.rpm
-rw-r--r--. 1 root root 12K Mar 29 14:31 python3-dnf-plugin-ulninfo-0.3-3.module+el8.10.0+90380+96a02ce9.noarch.rpm
-rw-r--r--. 1 root root 93K Mar 29 14:31 librepo-1.14.2-4.el8.x86_64.rpm
-rw-r--r--. 1 root root 40K Mar 29 14:43 python3-rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm

[root@ODA01 local_repo_issue_rhn]# dnf install *.rpm
BKB_Common                                                                                                                                                                                                                            37 kB/s | 1.5 kB     00:00
Dependencies resolved.
=====================================================================================================================================================================================================================================================================
 Package                                                              Architecture                                   Version                                                                              Repository                                            Size
=====================================================================================================================================================================================================================================================================
Installing:
 dnf-plugin-spacewalk                                                 noarch                                         2.8.5-11.0.3.module+el8.3.0+20070+f5719e00                                           @commandline                                          24 k
 python3-dnf-plugin-spacewalk                                         noarch                                         2.8.5-11.0.3.module+el8.3.0+20070+f5719e00                                           @commandline                                          31 k
 python3-dnf-plugin-ulninfo                                           noarch                                         0.3-3.module+el8.10.0+90380+96a02ce9                                                 @commandline                                          11 k
 python3-librepo                                                      x86_64                                         1.14.2-4.el8                                                                         @commandline                                          54 k
 python3-rhn-check                                                    x86_64                                         2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe                                          @commandline                                          39 k
 python3-rhn-setup                                                    x86_64                                         2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe                                          @commandline                                          71 k
 rhn-check                                                            x86_64                                         2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe                                          @commandline                                          26 k
 rhn-setup                                                            x86_64                                         2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe                                          @commandline                                          32 k
 rhnsd                                                                x86_64                                         5.0.35-3.0.1.module+el8+5192+3173336a                                                @commandline                                          51 k
Downgrading:
 librepo                                                              x86_64                                         1.14.2-4.el8                                                                         @commandline                                          93 k

Transaction Summary
=====================================================================================================================================================================================================================================================================
Install    9 Packages
Downgrade  1 Package

Total size: 432 k
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                             1/1
  Downgrading      : librepo-1.14.2-4.el8.x86_64                                                                                                                                                                                                                1/11
  Installing       : python3-librepo-1.14.2-4.el8.x86_64                                                                                                                                                                                                        2/11
  Running scriptlet: dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch                                                                                                                                                                     3/11
  Installing       : dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch                                                                                                                                                                     3/11
  Running scriptlet: dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch                                                                                                                                                                     3/11
  Installing       : python3-dnf-plugin-ulninfo-0.3-3.module+el8.10.0+90380+96a02ce9.noarch                                                                                                                                                                     4/11
  Installing       : python3-dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch                                                                                                                                                             5/11
  Installing       : python3-rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                       6/11
  Installing       : rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                               7/11
  Installing       : rhnsd-5.0.35-3.0.1.module+el8+5192+3173336a.x86_64                                                                                                                                                                                         8/11
  Running scriptlet: rhnsd-5.0.35-3.0.1.module+el8+5192+3173336a.x86_64                                                                                                                                                                                         8/11
  Installing       : python3-rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                       9/11
  Installing       : rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                              10/11
  Cleanup          : librepo-1.14.2-5.el8.x86_64                                                                                                                                                                                                               11/11
  Running scriptlet: librepo-1.14.2-5.el8.x86_64                                                                                                                                                                                                               11/11
  Verifying        : librepo-1.14.2-4.el8.x86_64                                                                                                                                                                                                                1/11
  Verifying        : librepo-1.14.2-5.el8.x86_64                                                                                                                                                                                                                2/11
  Verifying        : dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch                                                                                                                                                                     3/11
  Verifying        : python3-dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch                                                                                                                                                             4/11
  Verifying        : python3-dnf-plugin-ulninfo-0.3-3.module+el8.10.0+90380+96a02ce9.noarch                                                                                                                                                                     5/11
  Verifying        : python3-librepo-1.14.2-4.el8.x86_64                                                                                                                                                                                                        6/11
  Verifying        : python3-rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                       7/11
  Verifying        : python3-rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                       8/11
  Verifying        : rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                               9/11
  Verifying        : rhnsd-5.0.35-3.0.1.module+el8+5192+3173336a.x86_64                                                                                                                                                                                        10/11
  Verifying        : rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                              11/11

Downgraded:
  librepo-1.14.2-4.el8.x86_64
Installed:
  dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch              python3-dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch              python3-dnf-plugin-ulninfo-0.3-3.module+el8.10.0+90380+96a02ce9.noarch
  python3-librepo-1.14.2-4.el8.x86_64                                                 python3-rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                        python3-rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
  rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                        rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                rhnsd-5.0.35-3.0.1.module+el8+5192+3173336a.x86_64

Complete!
[root@ODA01 local_repo_issue_rhn]#

I checked the packages were there:

[root@ODA01 local_repo_issue_rhn]# rpm -qa | grep -i rhn
python3-rhnlib-2.8.6-8.0.2.module+el8.7.0+21027+f0093b7a.noarch
rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
python3-rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
python3-rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
rhnsd-5.0.35-3.0.1.module+el8+5192+3173336a.x86_64
python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
rhnlib-2.8.6-8.0.2.module+el8.7.0+21027+f0093b7a.noarch
[root@ODA01 local_repo_issue_rhn]#

And rerun a precheck report:

[root@ODA01 ~]# /opt/oracle/dcs/bin/odacli create-prepatchreport -sc -gi -st -v 19.30.0.0.0

Job details
----------------------------------------------------------------
                     ID:  3d5f5d96-f566-40c7-ad84-38c436fa3bba
            Description:  Patch pre-checks for [OS, ILOM, ORACHKSERVER, SERVER, STORAGE, RHPGI, GI] to 19.30.0.0.0
                 Status:  Created
                Created:  March 29, 2026 14:46:06 CEST
                Message:  Use 'odacli describe-prepatchreport -i 3d5f5d96-f566-40c7-ad84-38c436fa3bba' to check details of results

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------

[root@ODA01 ~]#

Which was that time successful:

[root@ODA01 ~]# odacli describe-prepatchreport -i 3d5f5d96-f566-40c7-ad84-38c436fa3bba

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  3d5f5d96-f566-40c7-ad84-38c436fa3bba
            Description:  Patch pre-checks for [OS, ILOM, ORACHKSERVER, SERVER, STORAGE, RHPGI, GI] to 19.30.0.0.0
                 Status:  SUCCESS
                Created:  March 29, 2026 2:46:06 PM CEST
                 Result:  All pre-checks succeeded

Node Name
---------------
ODA01

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is patch location available     Success   Patch location is available.
Verify All OS patches           Success   No dependencies found for RPMs being
                                          removed, updated and installed. Check
                                          /opt/oracle/dcs/log/jobfiles/
                                          dnfdryrunout_2026-03-29_14-46-
                                          18.0970_2312.log file for more details
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate all KVM ACFS           Success   All KVM ACFS resources are running
resources are running
Validate DB System VM states    Success   All DB System VMs states are expected
Validate command execution      Success   Validated command execution

__STORAGE__
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Patch location validation       Success   Verified patch location
Patch tag validation            Success   Verified patch tag
Storage patch tag validation    Success   Verified storage patch location
Verify ASM disks status         Success   ASM disks are online
Validate command execution      Success   Validated command execution

__RHPGI__
Validate available space        Success   Validated free space under /u01
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Successfully validated minimum version
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM is online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Validate space for clones       Success   Clones volume is already created
volume
Validate command execution      Success   Validated command execution

[root@ODA01 ~]#

I could then successfully patch the server components part:

[root@ODA01 ~]# /opt/oracle/dcs/bin/odacli update-servercomponents -v 19.30.0.0.0
{
  "jobId" : "fd6ed1d5-1982-46da-8c8c-dfa862f6427a",
  "status" : "Created",
  "message" : "A server update job has been initiated. If a reboot is required, it will be performed as part of the update process. The final status will be provided once the update operation has been completed on all nodes.",
  "reports" : [ ],
  "createTimestamp" : "March 29, 2026 15:00:09 CEST",
  "resourceList" : [ ],
  "description" : "Server Patching to 19.30.0.0.0",
  "updatedTime" : "March 29, 2026 15:00:09 CEST",
  "jobType" : null,
  "cpsMetadata" : null,
  "ecJobId" : null
}
[root@ODA01 ~]#


[root@ODA01 ~]# odacli describe-job -i fd6ed1d5-1982-46da-8c8c-dfa862f6427a

Job details
----------------------------------------------------------------
                     ID:  fd6ed1d5-1982-46da-8c8c-dfa862f6427a
            Description:  Server Patching to 19.30.0.0.0
                 Status:  Success
                Created:  March 29, 2026 15:00:09 CEST
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Modify BM udev rules                     March 29, 2026 15:00:14 CEST             March 29, 2026 15:00:26 CEST             Success
Validate ILOM server reachable           March 29, 2026 15:00:14 CEST             March 29, 2026 15:00:14 CEST             Success
Validating GI user metadata              March 29, 2026 15:00:14 CEST             March 29, 2026 15:00:14 CEST             Success
Stop oakd                                March 29, 2026 15:00:26 CEST             March 29, 2026 15:00:30 CEST             Success
Creating local repository                March 29, 2026 15:00:30 CEST             March 29, 2026 15:00:31 CEST             Success
OSPatchBaseRepo
Updating versionlock plugin              March 29, 2026 15:00:31 CEST             March 29, 2026 15:00:34 CEST             Success
Applying OS Patches                      March 29, 2026 15:00:34 CEST             March 29, 2026 15:07:16 CEST             Success
Applying HMP Patches                     March 29, 2026 15:07:16 CEST             March 29, 2026 15:07:19 CEST             Success
Creating local repository HMPPatchRepo   March 29, 2026 15:07:16 CEST             March 29, 2026 15:07:16 CEST             Success
Patch location validation                March 29, 2026 15:07:19 CEST             March 29, 2026 15:07:19 CEST             Success
Setting SELinux mode                     March 29, 2026 15:07:19 CEST             March 29, 2026 15:07:19 CEST             Success
Oda-hw-mgmt upgrade                      March 29, 2026 15:07:20 CEST             March 29, 2026 15:07:53 CEST             Success
Installing SQLcl software                March 29, 2026 15:07:53 CEST             March 29, 2026 15:07:57 CEST             Success
OSS Patching                             March 29, 2026 15:07:53 CEST             March 29, 2026 15:07:53 CEST             Success
Applying Firmware local Disk Patches     March 29, 2026 15:07:57 CEST             March 29, 2026 15:07:59 CEST             Success
Applying Firmware local Controller Patch March 29, 2026 15:07:59 CEST             March 29, 2026 15:11:23 CEST             Success
Applying Firmware shared Controller      March 29, 2026 15:11:23 CEST             March 29, 2026 15:11:42 CEST             Success
Patch
Checking Ilom patch Version              March 29, 2026 15:11:43 CEST             March 29, 2026 15:11:43 CEST             Success
Disabling IPMI v2                        March 29, 2026 15:11:43 CEST             March 29, 2026 15:11:44 CEST             Success
Patch location validation                March 29, 2026 15:11:43 CEST             March 29, 2026 15:11:43 CEST             Success
Save password in Wallet                  March 29, 2026 15:11:43 CEST             March 29, 2026 15:11:43 CEST             Success
Apply Ilom patch                         March 29, 2026 15:11:44 CEST             March 29, 2026 15:22:19 CEST             Success
Copying Flash Bios to Temp location      March 29, 2026 15:22:19 CEST             March 29, 2026 15:22:19 CEST             Success
Start oakd                               March 29, 2026 15:22:20 CEST             March 29, 2026 15:22:36 CEST             Success
Add SYSNAME in Env                       March 29, 2026 15:22:37 CEST             March 29, 2026 15:22:37 CEST             Success
Cleanup JRE Home                         March 29, 2026 15:22:37 CEST             March 29, 2026 15:22:37 CEST             Success
Starting the clusterware                 March 29, 2026 15:22:37 CEST             March 29, 2026 15:23:29 CEST             Success
Update lvm.conf file                     March 29, 2026 15:23:30 CEST             March 29, 2026 15:23:30 CEST             Success
Generating and saving BOM                March 29, 2026 15:23:31 CEST             March 29, 2026 15:25:37 CEST             Success
Update System full patch version         March 29, 2026 15:23:31 CEST             March 29, 2026 15:23:31 CEST             Success
Update System rebootless patch version   March 29, 2026 15:23:31 CEST             March 29, 2026 15:23:31 CEST             Success
PreRebootNode Actions                    March 29, 2026 15:25:37 CEST             March 29, 2026 15:26:16 CEST             Success
Reboot Node                              March 29, 2026 15:26:16 CEST             March 29, 2026 15:39:48 CEST             Success

[root@ODA01 ~]#

To wrap up…

ODA is an appliance. Any bad behaviour with the rpm and dependencies will bring life more difficult to the consultant coming to patch your ODA. So thanks for sparing a thought for the consultant who will have to patch your ODA before you start playing with yum or dnf on your ODA. 😉

L’article ODA 19.30 prepatch report failing on OS patches est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/oda-19-30-prepatch-report-failing-on-os-patches/feed/ 0 How to check if my Oracle dbhome is RO or RW? https://www.dbi-services.com/blog/how-to-check-if-my-oracle-dbhome-is-ro-or-rw/ https://www.dbi-services.com/blog/how-to-check-if-my-oracle-dbhome-is-ro-or-rw/#respond Fri, 10 Oct 2025 14:10:18 +0000 https://www.dbi-services.com/blog/?p=40899 As you might know, RO (Read Only) dbhome was introduced in Oracle 18c and became default configuration in Oracle 21c. Oracle came back with RW (Read Write) dbhome as default configuration in 23ai. It might be interesting to know how we can check if the oracle dbhome is RO (Read Only) or RW (Read Write). […]

L’article How to check if my Oracle dbhome is RO or RW? est apparu en premier sur dbi Blog.

]]> As you might know, RO (Read Only) dbhome was introduced in Oracle 18c and became default configuration in Oracle 21c. Oracle came back with RW (Read Write) dbhome as default configuration in 23ai. It might be interesting to know how we can check if the oracle dbhome is RO (Read Only) or RW (Read Write). I had to recently look for it and just wanted to share it through this blog.

Environment

For my test I used a FREE Oracle 23ai version. Here are my ORACLE_HOME and my ORACLE_BASE environment:

oracle@vmdmkdev:~/ [FREE (CDB$ROOT)] echo $ORACLE_HOME
/opt/oracle/product/23ai/dbhomeFree

oracle@vmdmkdev:~/ [FREE (CDB$ROOT)] echo $ORACLE_BASE
/opt/oracle

We know that for RW dbhome the dbs folder will be used in $ORACLE_HOME and for RO in $ORACLE_BASE.

ORACLE_BASE_HOME will be $ORACLE_HOME for RW and $ORACLE_BASE/homes/<HOME_NAME> for RO.

I did not activate RO on my dbhome, so I should not have so far any dbs or homes directory in my $ORACLE_BASE:

oracle@vmdmkdev:~/ [rdbms2300] ls $ORACLE_BASE
admin  audit  cfgtoollogs  diag  oraInventory  product

ROOH stands for Read Only Oracle HOME.

Activate ROOH

I will active ROOH for my dbhome.

oracle@vmdmkdev:~/ [rdbms2300] roohctl -enable
Enabling Read-Only Oracle home.
Cannot enable Read-Only Oracle home in a configured Oracle home.
The Oracle Home is configured with databases 'FREE'.

I still have my FREE instance database using the dbhome. I have just updated oratab file to comment the database.

oracle@vmdmkdev:~/ [rdbms2300] vi /etc/oratab

And ran roohctl command again, which this time will be successful.

oracle@vmdmkdev:~/ [rdbms2300] roohctl -enable
Enabling Read-Only Oracle home.
Update orabasetab file to enable Read-Only Oracle home.
Orabasetab file has been updated successfully.
Create bootstrap directories for Read-Only Oracle home.
Bootstrap directories have been created successfully.
Bootstrap files have been processed successfully.
Bootstrap files have been processed successfully.
Read-Only Oracle home has been enabled successfully.
Check the log file /opt/oracle/cfgtoollogs/roohctl/roohctl-251003PM023449.log for more details.

Now I have got a homes folder in $ORACLE_BASE as well as a dbs one.

oracle@vmdmkdev:~/ [rdbms2300] ls $ORACLE_BASE
admin  audit  cfgtoollogs  dbs  diag  homes  oraInventory  product

Contents of homes folder:

oracle@vmdmkdev:~/ [rdbms2300] ls $ORACLE_BASE/homes
OraDBHome23aiFree
oracle@vmdmkdev:~/ [rdbms2300] ls $ORACLE_BASE/homes/OraDBHome23aiFree/
assistants  dbs  drdaas  hs  install  mgw  network  rdbms
oracle@vmdmkdev:~/ [rdbms2300] ls $ORACLE_BASE/homes/OraDBHome23aiFree/dbs
oracle@vmdmkdev:~/ [rdbms2300]

My dbs folder in $ORACLE_HOME remains, of course, unchanged:

oracle@vmdmkdev:~/ [rdbms2300] cdh
oracle@vmdmkdev:/opt/oracle/product/23ai/dbhomeFree/ [rdbms2300] cd dbs
oracle@vmdmkdev:/opt/oracle/product/23ai/dbhomeFree/dbs/ [rdbms2300] ls
hc_FREE.dat  init.ora  lkFREE_SITE1  orapwFREE  spfileFREE.ora

Deactivate ROOH again

Let’s deactivate ROOH again and make the Oracle Home RW again to start all the tests.

oracle@vmdmkdev:/opt/oracle/product/23ai/dbhomeFree/dbs/ [rdbms2300] roohctl -disable
Disabling Read-Only Oracle home.
Update orabasetab file to disable Read-Only Oracle home.
Orabasetab file has been updated successfully.
Read-Only Oracle home has been disabled successfully.
Check the log file /opt/oracle/cfgtoollogs/roohctl/roohctl-251003PM024030.log for more details.

dbs folder in $ORACLE_BASE will remain unchanged.

oracle@vmdmkdev:/opt/oracle/product/23ai/dbhomeFree/dbs/ [rdbms2300] cd $ORACLE_BASE
oracle@vmdmkdev:/opt/oracle/ [rdbms2300] ls
admin  audit  cfgtoollogs  dbs  diag  homes  oraInventory  product

Check dbhome status with orabasehome binary

There is a binary file, orabasehome, stored in $ORACLE_HOME/bin which will tell me if my Oracle Home is RO or RW. It will just provide the value of ORACLE_BASE_HOME. So either $ORACLE_HOME folder if it is a RW dbhome or $ORACLE_BASE/homes/<HOME_NAME> folder if it is a RO dbhome.

oracle@vmdmkdev:/opt/oracle/dbs/ [rdbms2300] $ORACLE_HOME/bin/orabasehome
/opt/oracle/product/23ai/dbhomeFree
oracle@vmdmkdev:/opt/oracle/dbs/ [rdbms2300]

The binary file will display in my case $ORACLE_HOME, which is correct.

Start the database

I tested instance startup with file either in $ORACLE_HOME/dbs or $ORACLE_BASE/dbs folder.

oracle@vmdmkdev:/opt/oracle/dbs/ [FREE (CDB$ROOT)] sqh

SQL*Plus: Release 23.0.0.0.0 - for Oracle Cloud and Engineered Systems on Fri Oct 3 14:49:46 2025
Version 23.5.0.24.07

Copyright (c) 1982, 2024, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 1070721856 bytes
Fixed Size		    5368640 bytes
Variable Size		  335544320 bytes
Database Buffers	  725614592 bytes
Redo Buffers		    4194304 bytes
Database mounted.
Database opened.

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

Database is starting correctly.

I moved the instance file from $ORACLE_HOME/dbs to $ORACLE_BASE/dbs:

oracle@vmdmkdev:/opt/oracle/dbs/ [FREE (CDB$ROOT)] ls -ltrh $ORACLE_HOME/dbs
total 18M
-rw-r-----. 1 oracle oinstall 3.1K May 14  2015 init.ora
lrwxrwxrwx. 1 oracle oinstall   38 Sep 26  2024 orapwFREE -> /opt/oracle/admin/FREE/pfile/orapwFREE
-rw-r-----. 1 oracle oinstall   24 Sep 26  2024 lkFREE_SITE1
-rw-r-----. 1 oracle oinstall 4.5K Oct  3 14:49 spfileFREE.ora
-rw-r-----. 1 oracle oinstall  18M Oct  3 14:59 snapcf_FREE.f
-rw-rw----. 1 oracle oinstall 1.6K Oct  3 15:06 hc_FREE.dat

oracle@vmdmkdev:/opt/oracle/dbs/ [FREE (CDB$ROOT)] ls -ltrh $ORACLE_BASE/dbs
total 0
oracle@vmdmkdev:/opt/oracle/dbs/ [FREE (CDB$ROOT)] mv $ORACLE_HOME/dbs/* $ORACLE_BASE/dbs

oracle@vmdmkdev:/opt/oracle/dbs/ [FREE (CDB$ROOT)] ls -ltrh $ORACLE_HOME/dbs
total 0
oracle@vmdmkdev:/opt/oracle/dbs/ [FREE (CDB$ROOT)] ls -ltrh $ORACLE_BASE/dbs
total 18M
-rw-r-----. 1 oracle oinstall 3.1K May 14  2015 init.ora
lrwxrwxrwx. 1 oracle oinstall   38 Sep 26  2024 orapwFREE -> /opt/oracle/admin/FREE/pfile/orapwFREE
-rw-r-----. 1 oracle oinstall   24 Sep 26  2024 lkFREE_SITE1
-rw-r-----. 1 oracle oinstall 4.5K Oct  3 14:49 spfileFREE.ora
-rw-r-----. 1 oracle oinstall  18M Oct  3 14:59 snapcf_FREE.f
-rw-rw----. 1 oracle oinstall 1.6K Oct  3 15:06 hc_FREE.dat

Starting the database would fail:

oracle@vmdmkdev:/opt/oracle/dbs/ [FREE (CDB$ROOT)] sqh

SQL*Plus: Release 23.0.0.0.0 - for Oracle Cloud and Engineered Systems on Fri Oct 3 15:18:36 2025
Version 23.5.0.24.07

Copyright (c) 1982, 2024, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/opt/oracle/product/23ai/dbhomeFree/dbs/initFREE.ora'
SQL>

So Oracle is looking only for instance file in $ORACLE_HOME/dbs when configured in RW mode, which makes sense.

I moved database’s instance files again to $ORACLE_HOME/dbs, and delete $ORACLE_BASE/dbs and $ORACLE_BASE/homes.

oracle@vmdmkdev:/opt/oracle/ [FREE (CDB$ROOT)] mv $ORACLE_BASE/dbs/* $ORACLE_HOME/dbs/
oracle@vmdmkdev:/opt/oracle/ [FREE (CDB$ROOT)] ls $ORACLE_BASE/dbs/
oracle@vmdmkdev:/opt/oracle/ [FREE (CDB$ROOT)] ls $ORACLE_HOME/dbs/
hc_FREE.dat  init.ora  lkFREE_SITE1  orapwFREE  snapcf_FREE.f  spfileFREE.ora
oracle@vmdmkdev:/opt/oracle/ [FREE (CDB$ROOT)] rmdir $ORACLE_BASE/dbs/
oracle@vmdmkdev:/opt/oracle/ [FREE (CDB$ROOT)] rm -rf $ORACLE_BASE/homes

Activate ROOH again

I activated ROOH for the dbhome and checked dbs and homes folders to be created again.

oracle@vmdmkdev:~/ [FREE (CDB$ROOT)] roohctl -enable
Enabling Read-Only Oracle home.
Update orabasetab file to enable Read-Only Oracle home.
Orabasetab file has been updated successfully.
Create bootstrap directories for Read-Only Oracle home.
Bootstrap directories have been created successfully.
Bootstrap files have been processed successfully.
Bootstrap files have been processed successfully.
Read-Only Oracle home has been enabled successfully.
Check the log file /opt/oracle/cfgtoollogs/roohctl/roohctl-251003PM032952.log for more details.

oracle@vmdmkdev:~/ [FREE (CDB$ROOT)] ls -l $ORACLE_BASE | grep -iE 'dbs|homes'
drwxr-x---.  2 oracle oinstall    6 Oct 10 15:39 dbs
drwxr-x---.  3 oracle oinstall   31 Oct 10 15:39 homes
oracle@vmdmkdev:~/ [FREE (CDB$ROOT)]

Check dbhome status with orabasehome binary

Binary file is now showing $ORACLE_BASE/homes/HOME_NAME, so RO is really activated on my dbhome.

oracle@vmdmkdev:~/ [FREE (CDB$ROOT)] $ORACLE_HOME/bin/orabasehome
/opt/oracle/homes/OraDBHome23aiFree
oracle@vmdmkdev:~/ [FREE (CDB$ROOT)]

Start the database

Starting the database would fail, as there is no instance file in $ORACLE_BASE/dbs folder.

oracle@vmdmkdev:~/ [FREE (CDB$ROOT)] sqh

SQL*Plus: Release 23.0.0.0.0 - for Oracle Cloud and Engineered Systems on Fri Oct 3 15:30:55 2025
Version 23.5.0.24.07

Copyright (c) 1982, 2024, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/opt/oracle/dbs/initFREE.ora'
SQL>

As expected…

By the way, the inventory.xml file is not displaying anything about RW or RO.

Let’s move the instance files from $ORACLE_HOME/dbs to $ORACLE_BASE/dbs:

oracle@vmdmkdev:~/ [FREE (CDB$ROOT)] ls -l $ORACLE_BASE/dbs
total 0
oracle@vmdmkdev:~/ [FREE (CDB$ROOT)] ls -l $ORACLE_HOME/dbs
total 18340
-rw-rw----. 1 oracle oinstall     1544 Oct  3 15:06 hc_FREE.dat
-rw-r-----. 1 oracle oinstall     3079 May 14  2015 init.ora
-rw-r-----. 1 oracle oinstall       24 Sep 26  2024 lkFREE_SITE1
lrwxrwxrwx. 1 oracle oinstall       38 Sep 26  2024 orapwFREE -> /opt/oracle/admin/FREE/pfile/orapwFREE
-rw-r-----. 1 oracle oinstall 18759680 Oct  3 14:59 snapcf_FREE.f
-rw-r-----. 1 oracle oinstall     4608 Oct  3 14:49 spfileFREE.ora

oracle@vmdmkdev:~/ [FREE (CDB$ROOT)] mv $ORACLE_HOME/dbs/* $ORACLE_BASE/dbs

oracle@vmdmkdev:~/ [FREE (CDB$ROOT)] ls -l $ORACLE_BASE/dbs
total 18340
-rw-rw----. 1 oracle oinstall     1544 Oct  3 15:06 hc_FREE.dat
-rw-r-----. 1 oracle oinstall     3079 May 14  2015 init.ora
-rw-r-----. 1 oracle oinstall       24 Sep 26  2024 lkFREE_SITE1
lrwxrwxrwx. 1 oracle oinstall       38 Sep 26  2024 orapwFREE -> /opt/oracle/admin/FREE/pfile/orapwFREE
-rw-r-----. 1 oracle oinstall 18759680 Oct  3 14:59 snapcf_FREE.f
-rw-r-----. 1 oracle oinstall     4608 Oct  3 14:49 spfileFREE.ora
oracle@vmdmkdev:~/ [FREE (CDB$ROOT)] ls -l $ORACLE_HOME/dbs
total 0
oracle@vmdmkdev:~/ [FREE (CDB$ROOT)]

And database can be started:

oracle@vmdmkdev:~/ [FREE (CDB$ROOT)] sqh

SQL*Plus: Release 23.0.0.0.0 - for Oracle Cloud and Engineered Systems on Fri Oct 3 15:48:23 2025
Version 23.5.0.24.07

Copyright (c) 1982, 2024, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 1070721856 bytes
Fixed Size		    5368640 bytes
Variable Size		  335544320 bytes
Database Buffers	  725614592 bytes
Redo Buffers		    4194304 bytes
Database mounted.
Database opened.
SQL>

To wrap up…

If dbhome in RW or RO it looks in appropriate dbs directory, $ORACLE_HOME for RW and $ORACLE_BASE for RO.
Oracle inventory will not give us more details about if the Oracle dbhome is RW or RO.
orabasehome binary is the solution to know if the Oracle dbhome is in RW or RO.

L’article How to check if my Oracle dbhome is RO or RW? est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/how-to-check-if-my-oracle-dbhome-is-ro-or-rw/feed/ 0 Possible to relocate a PDB on ExaCC with move option? https://www.dbi-services.com/blog/possible-to-relocate-a-pdb-on-exacc-with-move-option/ https://www.dbi-services.com/blog/possible-to-relocate-a-pdb-on-exacc-with-move-option/#respond Fri, 10 Oct 2025 10:40:47 +0000 https://www.dbi-services.com/blog/?p=40870 In some of my previous blogs, I showed how we were migrating on-premise databases to the ExaCC using Oracle ZDM at one of our customer. I also explained in some article how it was important in some case to relocate the PDB in a final CDB. Relocating a PDB on ExaCC is usually done using […]

L’article Possible to relocate a PDB on ExaCC with move option? est apparu en premier sur dbi Blog.

]]> In some of my previous blogs, I showed how we were migrating on-premise databases to the ExaCC using Oracle ZDM at one of our customer. I also explained in some article how it was important in some case to relocate the PDB in a final CDB. Relocating a PDB on ExaCC is usually done using dbaascli pdb relocate command. The only drawback of this way, is that dbaascli is doing a copy of the datafiles during the relocate, which for a huge terabytes database, is taking time. In oder to minimise the downtime, I had to study another way to relocate the pdb, by moving the datafiles instead of doing a copy, and could manually relocate it in a few minutes. In this blog, I would like to share with you my findings and hoping it can help some of you.

Purpose

In my case, I would like to relocate PDB, named PDB_DRY_999T, from the source CDB, named CDB_CHZ2, into the target CDB, named CDBTGT_CHZ2.

Source CDB information:

oracle@exacc-cl01n1:~/ [CDB1 (CDB$ROOT)] CDB1

 ***********************************
 INSTANCE_NAME   : CDB1
 DB_NAME         : CDB
 DB_UNIQUE_NAME  : CDB_CHZ2
 STATUS          : OPEN READ WRITE
 LOG_MODE        : ARCHIVELOG
 USERS/SESSIONS  : 2/9
 DATABASE_ROLE   : PRIMARY
 FLASHBACK_ON    : YES
 FORCE_LOGGING   : YES
 VERSION         : 19.26.0.0.0
 CDB_ENABLED     : YES
 PDBs            : PDB_DRY_999T  PDB$SEED
 ***********************************

Target CDB information:

oracle@exacc-cl01n1:~/ [CDB1 (CDB$ROOT)] CDBTGT1

 ***********************************
 INSTANCE_NAME   : CDBTGT1
 DB_NAME         : CDBTGT
 DB_UNIQUE_NAME  : CDBTGT_CHZ2
 STATUS          : OPEN READ WRITE
 LOG_MODE        : ARCHIVELOG
 USERS/SESSIONS  : 2/10
 DATABASE_ROLE   : PRIMARY
 FLASHBACK_ON    : YES
 FORCE_LOGGING   : YES
 VERSION         : 19.26.0.0.0
 CDB_ENABLED     : YES
 PDBs            : PDB$SEED
 ***********************************

Knowing we are using TDE with ExaCC, it will be needed to export and import the encryption key. I’m using the FORCE option during export and import of the encryption key because I’m using AUTO LOGIN.

Export the encryption key on the PDB from the source CDB_CHZ2

Let’s connect to the PDB and check encryption.

oracle@exacc-cl01n1:~/ [CDB1 (CDB$ROOT)] sqh

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Aug 29 10:05:55 2025
Version 19.26.0.0.0

Copyright (c) 1982, 2024, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.26.0.0.0

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB_DRY_999T                   READ WRITE NO

SQL> alter session set container=PDB_DRY_999T;

Session altered.

SQL> set lines 300 pages 500
SQL> col WRL_PARAMETER format a30

SQL> select WRL_PARAMETER, WRL_TYPE,WALLET_TYPE, status from V$ENCRYPTION_WALLET;

WRL_PARAMETER                  WRL_TYPE             WALLET_TYPE          STATUS
------------------------------ -------------------- -------------------- ------------------------------
                               FILE                 AUTOLOGIN            OPEN

SQL> col status for a20
SQL> col name for a30

SQL> select a.con_id, name, status, keystore_mode from v$pdbs a, v$encryption_wallet b where a.con_id=b.con_id;

    CON_ID NAME                           STATUS               KEYSTORE
---------- ------------------------------ -------------------- --------
         3 PDB_DRY_999T                   OPEN                 UNITED

We are in united mode so all cdb and pdb encryption keys are stored in same keystore.

If we try to export the encryption key, we will get an ORA-46659 error:

SQL> ADMINISTER KEY MANAGEMENT EXPORT ENCRYPTION KEYS WITH SECRET "********************" to '/home/oracle/mwagner/TDE_for_unplug/PDB_DRY_999T_export.p12' FORCE keystore IDENTIFIED BY "********************";
ADMINISTER KEY MANAGEMENT EXPORT ENCRYPTION KEYS WITH SECRET "********************" to '/home/oracle/mwagner/TDE_for_unplug/PDB_DRY_999T_export.p12' FORCE keystore IDENTIFIED BY "********************"
*
ERROR at line 1:
ORA-46659: master keys for the given PDB not found

Let’s create a new master key for the PDB.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         3 PDB_DRY_999T                 READ WRITE NO

SQL> ADMINISTER KEY MANAGEMENT SET KEY FORCE KEYSTORE IDENTIFIED BY "********************" WITH BACKUP USING 'pre-unplug-PDB_DRY_999T_20250829';

keystore altered.

SQL>

And now we can export the pdb encryption key:

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         3 PDB_DRY_999T                   READ WRITE NO

SQL> ADMINISTER KEY MANAGEMENT EXPORT ENCRYPTION KEYS WITH SECRET "********************" to '/home/oracle/mwagner/TDE_for_unplug/PDB_DRY_999T_export_pdb_level.p12' FORCE keystore IDENTIFIED BY "********************";

keystore altered.

SQL>

Unplug PDB_DRY_999T PDB from source CDB

Let’s close the PDB and run ALTER PLUGGABLE DATABASE <PDB_NAME> UNPLUG INTO '<XML_FILE>' command:

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
CDB1

SQL> show con_name

CON_NAME
------------------------------
CDB$ROOT

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB_DRY_999T                   READ WRITE NO

SQL> alter pluggable database PDB_DRY_999T close instances=all;

Pluggable database altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB_DRY_999T                   MOUNTED

SQL> ALTER PLUGGABLE DATABASE PDB_DRY_999T UNPLUG INTO '/home/oracle/mwagner/unplug_pdb/PDB_DRY_999T.xml';

Pluggable database altered.

SQL> !ls -ltrh /home/oracle/mwagner/unplug_pdb/PDB_DRY_999T.xml
-rw-r--r-- 1 oracle asmdba 2.7M Aug 29 10:50 /home/oracle/mwagner/unplug_pdb/PDB_DRY_999T.xml

SQL>

In the alert log we would see some warnings to inform that the encryption key will need to be imported. Otherwise the pdb can only be opened in restricted mode:

PDB_DRY_999T(3):KZTDE:WARNING: Detected that PDB needs to import keys from source. PDB can only open in restricted mode until import.

This is of course makes sense.

Drop the PDB but keep datafiles!!!

As we are going to relocate the PDB moving the datafiles, it is VERY IMPORTANT to drop the PDB by KEEPING the datafiles.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB_DRY_999T                   MOUNTED

SQL> drop pluggable database PDB_DRY_999T keep datafiles;

Pluggable database dropped.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
SQL>

From the alert log file we can see that only the temporary file are dropped. This file will be recreated again by the oracle plug command.

2025-08-29T10:57:26.137914+02:00
Deleted Oracle managed file +DATAC2/CDB_CHZ2/3D6E40CA5F218D4EE063181FA10A38B0/TEMPFILE/temp.26231.1210348651
2025-08-29T10:57:26.168729+02:00
Stopped service pdb_dry_999t

If we connect to the ASM, we will see that the datafiles are still present. I still have 7.45 TB of datafiles for my PDB.

ASMCMD> du PDB_DRY_999T
Used_MB      Mirror_used_MB
7809404            23428212
ASMCMD>

Check compatibility for plugging the PDB in the target CDB

I will now connect to the target CDB, and check if the PDB is compatible to be plug in.

I will run following SQL command to do so:

set serveroutput on
DECLARE
compatible BOOLEAN := FALSE;
BEGIN
compatible := DBMS_PDB.CHECK_PLUG_COMPATIBILITY(
pdb_descr_file => '/home/oracle/mwagner/unplug_pdb/PDB_DRY_999T.xml');
if compatible then
DBMS_OUTPUT.PUT_LINE('Is pluggable PDB_DRY_999T compatible? YES');
else DBMS_OUTPUT.PUT_LINE('Is pluggable PDB_DRY_999T compatible? NO');
end if;
END;

Let’s connect to the target CDB and check:

oracle@exacc-cl01n1:~/ [CDB1 (CDB$ROOT)] CDBTGT1

 ***********************************
 INSTANCE_NAME   : CDBTGT1
 DB_NAME         : CDBTGT
 DB_UNIQUE_NAME  : CDBTGT_CHZ2
 STATUS          : OPEN READ WRITE
 LOG_MODE        : ARCHIVELOG
 USERS/SESSIONS  : 2/10
 DATABASE_ROLE   : PRIMARY
 FLASHBACK_ON    : YES
 FORCE_LOGGING   : YES
 VERSION         : 19.26.0.0.0
 CDB_ENABLED     : YES
 PDBs            : PDB$SEED
 ***********************************

 PDB color: pdbname=mount, pdbname=open read-write, pdbname=open read-only
 Statustime: 2025-08-29 11:07:00

oracle@exacc-cl01n1:~/ [CDBTGT1 (CDB$ROOT)] sqh

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Aug 29 11:07:02 2025
Version 19.26.0.0.0

Copyright (c) 1982, 2024, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.26.0.0.0

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO

SQL> set serveroutput on
SQL> DECLARE
  2  compatible BOOLEAN := FALSE;
  3  BEGIN
  4  compatible := DBMS_PDB.CHECK_PLUG_COMPATIBILITY(
  5  pdb_descr_file => '/home/oracle/mwagner/unplug_pdb/PDB_DRY_999T.xml');
  6  if compatible then
  7  DBMS_OUTPUT.PUT_LINE('Is pluggable PDB_DRY_999T compatible? YES');
  8  else DBMS_OUTPUT.PUT_LINE('Is pluggable PDB_DRY_999T compatible? NO');
  9  end if;
 10  END;
 11  /
Is pluggable PDB_DRY_999T compatible? YES

PL/SQL procedure successfully completed.

SQL>

The PDB can be plug in my target CDB.

Import PDB encryption key in the target CDB keystore

We first need to import the PDB encryption key in the CDB, otherwise the plugging command would failed with ORA-28374 error:

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
CDBTGT1

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO

SQL> show con_name

CON_NAME
------------------------------
CDB$ROOT

SQL> create pluggable database PDB_DRY_999T using '/home/oracle/mwagner/unplug_pdb/PDB_DRY_999T.xml' move;
create pluggable database PDB_DRY_999T using '/home/oracle/mwagner/unplug_pdb/PDB_DRY_999T.xml' move
*
ERROR at line 1:
ORA-28374: typed master key not found in wallet

So let’s import the PDB encryption key into the CDB:

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
CDBTGT1

SQL> show con_name

CON_NAME
------------------------------
CDB$ROOT

SQL> ADMINISTER KEY MANAGEMENT IMPORT ENCRYPTION KEYS WITH SECRET "********************" from '/home/oracle/mwagner/TDE_for_unplug/PDB_DRY_999T_export_pdb_level.p12' FORCE keystore IDENTIFIED BY "********************" with backup using 'pre-plug-PDB_DRY_999T_20250829';

keystore altered.

SQL>

Plug the PDB in the target CDB CDBTGT_CHZ2

Now we can plug the PDB in the target CDB:

SQL> !date
Fri Aug 29 11:27:36 CEST 2025

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
CDBTGT1

SQL> create pluggable database PDB_DRY_999T using '/home/oracle/mwagner/unplug_pdb/PDB_DRY_999T.xml' move;

Pluggable database created.

SQL> !date
Fri Aug 29 11:30:36 CEST 2025

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB_DRY_999T                   MOUNTED
SQL>

We can see that instead of taking several hours to relocate the PDB with dbaascli, we took about 3 minutes to relocate the PDB moving the datafiles.

We can check the alert log file for any errors. In my case, I can see the completed operation:

Completed: create pluggable database PDB_DRY_999T using '/home/oracle/mwagner/unplug_pdb/PDB_DRY_999T.xml' move
2025-08-29T11:30:52.168375+02:00

Checking the ASM I can see that the source datafile directory is now empty:

ASMCMD> pwd
+DATAC2

ASMCMD> du PDB_DRY_999T
Used_MB      Mirror_used_MB
  48780              146340

ASMCMD> cd PDB_DRY_999T

ASMCMD> ls
CHANGETRACKING/
DATAGUARDCONFIG/
TEMPFILE/
controlfile/
datafile/
onlinelog/
password/

ASMCMD> cd datafile

ASMCMD> pwd
+DATAC2/PDB_DRY_999T/datafile

ASMCMD> ls
ASMCMD>

Start the PDB

Let’s start the PDB.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB_DRY_999T                   MOUNTED

SQL> alter pluggable database PDB_DRY_999T open instances=all;

Warning: PDB altered with errors.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB_DRY_999T                   READ WRITE YES

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB_DRY_999T                   READ WRITE YES

The PDB is started in restricted mode. This was given as warning in the alert log during the unplug, by informing that the encryption key needs to be imported in the PDB in order to open it without any restriction.

Let’s check the violations.

SQL> select name,cause,type,message,status from PDB_PLUG_IN_VIOLATIONS where status  'RESOLVED';

NAME            CAUSE                                              TYPE      MESSAGE                                                                                                                  STATUS
--------------- -------------------------------------------------- --------- ------------------------------------------------------------------------------------------------------------------------ --------------------
PDB_DRY_999T  Parameter                                          WARNING   CDB parameter processes mismatch: Previous 2048 Current 1000                                                             PENDING
PDB_DRY_999T  Parameter                                          WARNING   CDB parameter sga_max_size mismatch: Previous 4G Current 9G                                                              PENDING
PDB_DRY_999T  Parameter                                          WARNING   CDB parameter archive_lag_target mismatch: Previous 1800 Current 0                                                       PENDING
PDB_DRY_999T  Parameter                                          WARNING   CDB parameter cluster_database mismatch: Previous FALSE Current TRUE                                                     PENDING
PDB_DRY_999T  Parameter                                          WARNING   CDB parameter db_lost_write_protect mismatch: Previous 'typical' Current 'NONE'                                          PENDING
PDB_DRY_999T  Parameter                                          WARNING   CDB parameter inmemory_force mismatch: Previous 'cellmemory_level' Current 'DEFAULT'                                     PENDING
PDB_DRY_999T  Parameter                                          WARNING   CDB parameter wallet_root mismatch: Previous '/var/opt/oracle/dbaas_acfs/CDB/wallet_root' Current '/var/opt/oracle/dbaa PENDING
                                                                             s_acfs/CDBTGT/wallet_root'

PDB_DRY_999T  Parameter                                          WARNING   CDB parameter distributed_lock_timeout mismatch: Previous 360 Current 60                                                 PENDING
PDB_DRY_999T  Parameter                                          WARNING   CDB parameter unified_audit_common_systemlog mismatch: Previous 'LOCAL5.INFO' Current NULL                               PENDING
PDB_DRY_999T  Parameter                                          WARNING   CDB parameter enable_ddl_logging mismatch: Previous TRUE Current FALSE                                                   PENDING
PDB_DRY_999T  Parameter                                          WARNING   CDB parameter awr_pdb_max_parallel_slaves mismatch: Previous 2 Current 10                                                PENDING
PDB_DRY_999T  Parameter                                          WARNING   CDB parameter awr_snapshot_time_offset mismatch: Previous 1000000 Current 0                                              PENDING
PDB_DRY_999T  OPTION                                             WARNING   Database option CONTEXT mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.                          PENDING
PDB_DRY_999T  OPTION                                             WARNING   Database option DV mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.                               PENDING
PDB_DRY_999T  OPTION                                             WARNING   Database option OLS mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.                              PENDING
PDB_DRY_999T  OPTION                                             WARNING   Database option ORDIM mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.                            PENDING
PDB_DRY_999T  OPTION                                             WARNING   Database option OWM mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.                              PENDING
PDB_DRY_999T  OPTION                                             WARNING   Database option SDO mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.                              PENDING
PDB_DRY_999T  Wallet Key Needed                                  ERROR     PDB needs to import keys from source.                                                                                    PENDING

19 rows selected.

SQL>

Here we have:

PDB_DRY_999T Wallet Key Needed ERROR PDB needs to import keys from source.

Import encryption key into the PDB

Let’s connect to the PDB and import the encryption key:

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB_DRY_999T                   READ WRITE YES

SQL> alter session set container=PDB_DRY_999T;

Session altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         3 PDB_DRY_999T                   READ WRITE YES

SQL> ADMINISTER KEY MANAGEMENT IMPORT ENCRYPTION KEYS WITH SECRET "********************" from '/home/oracle/mwagner/TDE_for_unplug/PDB_DRY_999T_export_pdb_level.p12' FORCE keystore IDENTIFIED BY "********************" with backup using 'post-plug-PDB_DRY_999T_20250829';

keystore altered.

SQL>

Restart the PDB

Let’s restart the PDB and check the status.

SQL> alter session set container=cdb$root;

Session altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB_DRY_999T                   READ WRITE YES

SQL> alter pluggable database PDB_DRY_999T close instances=all;

Pluggable database altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB_DRY_999T                   MOUNTED

SQL> alter pluggable database PDB_DRY_999T open instances=all;

Pluggable database altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB_DRY_999T                   READ WRITE NO
SQL>

We can see that the PDB is now opened without any restriction. Let’s check the PDB violations:

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         3 PDB_DRY_999T                   READ WRITE NO

SQL> col name for a15
SQL> col message for a120
SQL> col cause for a50

SQL> select name,cause,type,message,status from PDB_PLUG_IN_VIOLATIONS where status  'RESOLVED';

NAME            CAUSE                                              TYPE      MESSAGE                                                                                                                  STATUS
--------------- -------------------------------------------------- --------- ------------------------------------------------------------------------------------------------------------------------ --------------------
PDB_DRY_999T  OPTION                                             WARNING   Database option CONTEXT mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.                          PENDING
PDB_DRY_999T  OPTION                                             WARNING   Database option DV mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.                               PENDING
PDB_DRY_999T  OPTION                                             WARNING   Database option OLS mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.                              PENDING
PDB_DRY_999T  OPTION                                             WARNING   Database option ORDIM mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.                            PENDING
PDB_DRY_999T  OPTION                                             WARNING   Database option OWM mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.                              PENDING
PDB_DRY_999T  OPTION                                             WARNING   Database option SDO mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.                              PENDING

6 rows selected.

SQL>

There is no more violations.

Check the PDB

Check datafiles location:

SQL> select name from v$datafile where rownum select count(*) from v$datafile where name like '+DATAC2/CDBTGT_CHZ2/3D6E40CA5F218D4EE063181FA10A38B0/DATAFILE%';

  COUNT(*)
----------
      2830

SQL> select count(*) from v$datafile where name not like '+DATAC2/CDBTGT_CHZ2/3D6E40CA5F218D4EE063181FA10A38B0/DATAFILE%';

  COUNT(*)
----------
         0

SQL>

All good, PDB datafiles are in the expected ASM folder.

Check that there is no invalid objects:

SQL> select count(*) from dba_invalid_objects;

  COUNT(*)
----------
         0

And finally check that all tablespace are encrypted:

SQL> select count(*) from  cdb_tablespaces a, (select x.con_id, y.ENCRYPTIONALG, x.name from V$TABLESPACE x,  V$ENCRYPTED_TABLESPACES y
  2    where x.ts#=y.ts# and x.con_id=y.con_id) b where a.con_id=b.con_id(+) and a.tablespace_name=b.name(+) and b.ENCRYPTIONALG='AES128';

  COUNT(*)
----------
      2831

SQL> select count(*) from  cdb_tablespaces a, (select x.con_id, y.ENCRYPTIONALG, x.name from V$TABLESPACE x,  V$ENCRYPTED_TABLESPACES y
  2    where x.ts#=y.ts# and x.con_id=y.con_id) b where a.con_id=b.con_id(+) and a.tablespace_name=b.name(+) and b.ENCRYPTIONALG is null;

  COUNT(*)
----------
         0

SQL>

The PDB is as well registered in the listener:

oracle@exacc-cl01n1:~/ [CDBTGT1 (CDB$ROOT)] export TNS_ADMIN=/u02/app/oracle/product/19.0.0.0/dbhome_1/network/admin/listener_test/
oracle@exacc-cl01n1:~/ [CDBTGT1 (CDB$ROOT)] lsnrctl status LISTENER_TEST| grep -i PDB_DRY_999T
Service "pdb_dry_999t.domain.com" has 1 instance(s).
oracle@exacc-cl01n1:~/ [CDBTGT1 (CDB$ROOT)]

To wrap up

Here we have a good alternative for relocating a PDB on ExaCC avoiding a copy of the datafiles. This will have the benefit to limit the maintenance windows.

L’article Possible to relocate a PDB on ExaCC with move option? est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/possible-to-relocate-a-pdb-on-exacc-with-move-option/feed/ 0 ZDM 21.5 Physical Online Migration failing during TEMP tablespace encryption https://www.dbi-services.com/blog/zdm-21-5-physical-online-migration-failing-during-temp-tablespace-encryption/ https://www.dbi-services.com/blog/zdm-21-5-physical-online-migration-failing-during-temp-tablespace-encryption/#respond Tue, 05 Aug 2025 12:05:06 +0000 https://www.dbi-services.com/blog/?p=39954 The ZDM 21.5 version brings some new functions like encrypting automatically now the TEMP, SYSTEM, SYSAUX and UNDO tablespaces. There is no need to do it manually any more after the ZDM migration, ZDM will take care of it on its own. Little problem, this is adding a new bug because the temporary database needs […]

L’article ZDM 21.5 Physical Online Migration failing during TEMP tablespace encryption est apparu en premier sur dbi Blog.

]]> The ZDM 21.5 version brings some new functions like encrypting automatically now the TEMP, SYSTEM, SYSAUX and UNDO tablespaces. There is no need to do it manually any more after the ZDM migration, ZDM will take care of it on its own. Little problem, this is adding a new bug because the temporary database needs to be restarted to drop the previous TEMP tablespace. Let’s see how to resolve the problem and move forward with the ZDM migration.

Problem description

After switchover is run by ZDM, the TEMP, SYSTEM, SYSAUX and UNDO tablespaces are now encrypted by ZDM during the ZDM_POST_DATABASE_OPEN_TGT step. In my case described here, I was doing a dry run, so I manually ran a failover to the target environment to avoid any impact and downtime to the source database. It was currently a dry run and not the final migration. Thus, after having paused the ZDM migration once the Data Guard configuration was setup (ZDM_CONFIGURE_DG_SRC step), I manually ran the failover and then moved forward with the ZDM Migration using the -skip SWITCHOVER option. I have explained this in one of my previous ZDM blog if you want more details.

I’m running same kind of migration I was doing with 21.4 version, non-cdb as source and including conversion to pdb during ZDM migration.

So, after failover has been run manually, I resumed the ZDM migration till the end.

[exauser@zdm-host ~]$ zdmcli resume job -jobid 580 -skip SWITCHOVER
zdm-host.domain.com: Audit ID: 6963

As we can see, the ZDM job failed on the ZDM_POST_DATABASE_OPEN_TGT step.

[exauser@zdm-host ~]$ zdmcli query job -jobid 580
zdm-host.domain.com: Audit ID: 6966
Job ID: 580
User: exauser
Client: zdm-host
Job Type: "MIGRATE"
Scheduled job command: "zdmcli migrate database -sourcesid SRCSID -rsp /home/exauser/migration/zdm_SRCSID_physical_online.rsp -sourcenode dbsource-host -srcauth dbuser -srcarg1 user:oracle -srcarg2 identity_file:/home/exauser/.ssh/id_rsa -targetnode exacc-cl01n1 -tgtauth zdmauth -tgtarg1 user:opc -tgtarg2 identity_file:/home/exauser/.ssh/id_rsa -tgtarg3 sudo_location:/usr/bin/sudo -tdekeystorepasswd -tgttdekeystorepasswd -pauseafter ZDM_COPYFILES"
Scheduled job execution start time: 2025-06-11T17:09:40+02. Equivalent local time: 2025-06-11 17:09:40
Current status: FAILED
Result file path: "/u01/app/oracle/chkbase/scheduled/job-580-2025-06-11-17:10:06.log"
Metrics file path: "/u01/app/oracle/chkbase/scheduled/job-580-2025-06-11-17:10:06.json"
Job execution start time: 2025-06-11 17:10:07
Job execution end time: 2025-06-12 10:04:40
Job execution elapsed time: 2 hours 52 minutes 48 seconds
ZDM_GET_SRC_INFO .............. COMPLETED
ZDM_GET_TGT_INFO .............. COMPLETED
ZDM_PRECHECKS_SRC ............. COMPLETED
ZDM_PRECHECKS_TGT ............. COMPLETED
ZDM_SETUP_SRC ................. COMPLETED
ZDM_SETUP_TGT ................. COMPLETED
ZDM_PREUSERACTIONS ............ COMPLETED
ZDM_PREUSERACTIONS_TGT ........ COMPLETED
ZDM_VALIDATE_SRC .............. COMPLETED
ZDM_VALIDATE_TGT .............. COMPLETED
ZDM_DISCOVER_SRC .............. COMPLETED
ZDM_COPYFILES ................. COMPLETED
ZDM_PREPARE_TGT ............... COMPLETED
ZDM_SETUP_TDE_TGT ............. COMPLETED
ZDM_RESTORE_TGT ............... COMPLETED
ZDM_RECOVER_TGT ............... COMPLETED
ZDM_FINALIZE_TGT .............. COMPLETED
ZDM_CONFIGURE_DG_SRC .......... COMPLETED
ZDM_SWITCHOVER_SRC ............ COMPLETED
ZDM_SWITCHOVER_TGT ............ COMPLETED
ZDM_POST_DATABASE_OPEN_TGT .... FAILED
ZDM_DATAPATCH_TGT ............. PENDING
ZDM_NONCDBTOPDB_PRECHECK ...... PENDING
ZDM_NONCDBTOPDB_CONVERSION .... PENDING
ZDM_POST_MIGRATE_TGT .......... PENDING
ZDM_POSTUSERACTIONS ........... PENDING
ZDM_POSTUSERACTIONS_TGT ....... PENDING
ZDM_CLEANUP_SRC ............... PENDING
ZDM_CLEANUP_TGT ............... PENDING
[exauser@zdm-host ~]$

ZDM log file would show the problem coming from the TEMP tablespace encryption.

[exauser@zdm-host ~]$ tail -f /u01/app/oracle/chkbase/scheduled/job-580-2025-06-11-17:10:06.log
####################################################################
zdm-host: 2025-06-12T08:01:48.268Z : Resuming zero downtime migrate operation ...
zdm-host: 2025-06-12T08:01:52.847Z : Skipping phase ZDM_PRECHECKS_SRC on resume
zdm-host: 2025-06-12T08:01:53.255Z : Skipping phase ZDM_PRECHECKS_TGT on resume
zdm-host: 2025-06-12T08:01:53.273Z : Executing phase ZDM_SETUP_SRC
zdm-host: 2025-06-12T08:01:53.274Z : Setting up ZDM on the source node dbsource-host ...
dbsource-host: 2025-06-12T08:02:23.476Z : TNS aliases successfully setup on the source node source-hostname...
dbsource-host: 2025-06-12T08:02:34.101Z : successfully registered ZDM with the database SRCSID
zdm-host: 2025-06-12T08:02:34.106Z : Execution of phase ZDM_SETUP_SRC completed
zdm-host: 2025-06-12T08:02:34.123Z : Executing phase ZDM_SETUP_TGT
zdm-host: 2025-06-12T08:02:34.123Z : Setting up ZDM on the target node exacc-cl01n1 ...
exacc-cl01n1: 2025-06-12T08:03:00.739Z : TNS aliases successfully setup on the target node exacc-cl01n1...
zdm-host: 2025-06-12T08:03:00.742Z : Execution of phase ZDM_SETUP_TGT completed
zdm-host: 2025-06-12T08:03:00.754Z : Skipping phase ZDM_VALIDATE_SRC on resume
zdm-host: 2025-06-12T08:03:00.765Z : Skipping phase ZDM_VALIDATE_TGT on resume
zdm-host: 2025-06-12T08:03:00.779Z : Skipping phase ZDM_DISCOVER_SRC on resume
zdm-host: 2025-06-12T08:03:00.789Z : Skipping phase ZDM_COPYFILES on resume
zdm-host: 2025-06-12T08:03:00.802Z : Skipping phase ZDM_PREPARE_TGT on resume
zdm-host: 2025-06-12T08:03:00.813Z : Skipping phase ZDM_SETUP_TDE_TGT on resume
zdm-host: 2025-06-12T08:03:00.823Z : Skipping phase ZDM_RESTORE_TGT on resume
zdm-host: 2025-06-12T08:03:00.833Z : Skipping phase ZDM_RECOVER_TGT on resume
zdm-host: 2025-06-12T08:03:00.843Z : Skipping phase ZDM_FINALIZE_TGT on resume
zdm-host: 2025-06-12T08:03:00.844Z : resuming job execution from phase "ZDM_CONFIGURE_DG_SRC"
zdm-host: 2025-06-12T08:03:00.859Z : Skipping phase ZDM_CONFIGURE_DG_SRC on resume
zdm-host: 2025-06-12T08:03:00.880Z : Executing phase ZDM_SWITCHOVER_SRC
zdm-host: 2025-06-12T08:03:00.881Z : Switching database null on the source node dbsource-host to standby role ...
dbsource-host: 2025-06-12T08:03:11.403Z : Switchover actions in the source environment executed successfully
zdm-host: 2025-06-12T08:03:11.407Z : Execution of phase ZDM_SWITCHOVER_SRC completed
####################################################################
zdm-host: 2025-06-12T08:03:11.445Z : Executing phase ZDM_SWITCHOVER_TGT
zdm-host: 2025-06-12T08:03:11.445Z : Switching database SRCSID_CHZ1 on the target node exacc-cl01n1 to primary role ...
exacc-cl01n1: 2025-06-12T08:03:23.137Z : Switchover actions in the target environment executed successfully
zdm-host: 2025-06-12T08:03:23.241Z : Execution of phase ZDM_SWITCHOVER_TGT completed
####################################################################
zdm-host: 2025-06-12T08:03:23.279Z : Executing phase ZDM_POST_DATABASE_OPEN_TGT
zdm-host: 2025-06-12T08:03:23.280Z : Executing post database open actions at the target ...
exacc-cl01n1: 2025-06-12T08:03:34.667Z : Restoring pluggable database state ...
exacc-cl01n1: 2025-06-12T08:03:35.368Z : Creating SSO for keystore location /u02/app/oracle/zdm/zdm_SRCSID_CHZ1_580/zdm/TDE/tde/ ...
exacc-cl01n1: 2025-06-12T08:03:35.569Z : Restoring RMAN configuration ...
exacc-cl01n1: 2025-06-12T08:03:40.374Z : Encrypting unencrypted temp tablespaces...
####################################################################
zdm-host: 2025-06-12T08:04:40.845Z : Oracle ZDM ONLINE PHYSICAL migration failed
zdm-host: 2025-06-12T08:04:40.845Z : Failed at phase: ZDM_POST_DATABASE_OPEN_TGT
zdm-host: 2025-06-12T08:04:40.846Z : Job duration: 2 minutes and 56 seconds
PRGZ-3727 : Re-creation of temporary tablespace "TEMP" as encrypted for database "PDB001P" failed.

Troubleshooting

Let’s investigate the issue.

The ZDM step log, named zdm_post_database_open_tgt, would show exactly the issue and the command that are expected to be run.

oracle@exacc-cl01n1:/u02/app/oracle/zdm/zdm_SRCSID_CHZ1_580/zdm/log/ [PDB001P1 (CDB$ROOT)] view zdm_post_database_open_tgt_7924.log
...
...
...
[jobid-580][2025-06-12T08:03:41Z][mZDM_Queries.pm:655]:[DEBUG] Will be running following sql statements as user: oracle:

        export ORACLE_HOME=/u02/app/oracle/product/19.0.0.0/dbhome_1,
        export ORACLE_SID=PDB001P1,
        /u02/app/oracle/product/19.0.0.0/dbhome_1/bin/sqlplus -L / as sys ******

        set pagesize 0 feedback off verify off heading off echo off tab off linesize 32767 trimspool on trimout on wrap off
CREATE BIGFILE TEMPORARY TABLESPACE TEMP_ENCRYPTED TEMPFILE ENCRYPTION ENCRYPT
ALTER DATABASE DEFAULT TEMPORARY TABLESPACE TEMP_ENCRYPTED
DROP TABLESPACE TEMP INCLUDING CONTENTS
ALTER TABLESPACE TEMP_ENCRYPTED RENAME TO TEMP

[jobid-580][2025-06-12T08:03:41Z][mZDM_Utils.pm:3438]:[DEBUG] run_as_user2: Running /*******@/u02/app/oracle/zdm/zdm_SRCSID_CHZ1_580/zdm/out/sql/tmpsql_2025-06-12-08-03-41_7924.sql '
[jobid-580][2025-06-12T08:04:41Z][mZDM_Utils.pm:3460]:[DEBUG] Remove /tmp/xE6e9fcUzC
[jobid-580][2025-06-12T08:04:41Z][mZDM_Utils.pm:3468]:[DEBUG] /bin/su successfully executed

[jobid-580][2025-06-12T08:04:41Z][mZDM_Queries.pm:678]:[DEBUG] Output is :
 SQL*Plus: Release 19.0.0.0.0 - Production on Thu Jun 12 08:03:41 2025
 Version 19.26.0.0.0

 Copyright (c) 1982, 2024, Oracle.  All rights reserved.


 Connected to:
 Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
 Version 19.26.0.0.0

 DROP TABLESPACE TEMP INCLUDING CONTENTS
 *
 ERROR at line 1:
 ORA-60100: dropping temporary tablespace with tablespace ID number (tsn) 3 is blocked due to sort segments


 ALTER TABLESPACE TEMP_ENCRYPTED RENAME TO TEMP
 *
 ERROR at line 1:
 ORA-02154: a tablespace with the name 'TEMP' is found


 Disconnected from Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
 Version 19.26.0.0.0

[jobid-580][2025-06-12T08:04:41Z][mZDM_Queries.pm:11791]:[ERROR] SQL Error recreating temp tablespace TEMP as encrypted for database 'PDB001P'
[jobid-580][2025-06-12T08:04:41Z][mZDM_Utils.pm:4632]:[ERROR] Reporting  error :

The TEMP tablespace can not be dropped, the database needs to be restarted.

Resolution

Let’s first create a pfile for the temporary non-CDB database ZDM created for the migration. Reminder, this temporary database will have as instance name and db_unique_name the name of the final PDB. And of course as DB_NAME the same than our source database, mandatory, when using HA primary and standby databases. I will create a pfile just in case…

I created a pfile from memory:

oracle@exacc-cl01n1:/u02/app/oracle/zdm/zdm_SRCSID_CHZ1_580/zdm/log/ [PDB001P1 (CDB$ROOT)] sqh

SQL*Plus: Release 19.0.0.0.0 - Production on Thu Jun 12 10:14:20 2025
Version 19.26.0.0.0

Copyright (c) 1982, 2024, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.26.0.0.0

SQL> create pfile='/u02/app/oracle/zdm/zdm_SRCSID_CHZ1_580/zdm/log/init_PDB001P1.ora.bck.before.restart' from memory;

File created.

And a pfile from spfile:

SQL> show parameter spfile

NAME                                 TYPE                              VALUE
------------------------------------ --------------------------------- ------------------------------
spfile                               string                            /u02/app/oracle/product/19.0.0
                                                                       .0/dbhome_1/dbs/spfileAVQPZ_AP
                                                                       P_001P1.ora

SQL> create pfile='/u02/app/oracle/zdm/zdm_SRCSID_CHZ1_580/zdm/log/init_PDB001P1.ora.bck.before.restart.from.spfile' from spfile='/u02/app/oracle/product/19.0.0.0/dbhome_1/dbs/spfilePDB001P1.ora';

File created.

SQL>

I restarted the temporary non-CDB

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> startup
ORACLE instance started.

Total System Global Area 8577884400 bytes
Fixed Size                 13693168 bytes
Variable Size            3439329280 bytes
Database Buffers         5100273664 bytes
Redo Buffers               24588288 bytes
Database mounted.
ORA-28374: typed master key not found in wallet

ORA-28374 Failure. Would be the same if I start with the pfile, so there was some missing parameter into the spfile.

Confirmed, there is no wallet_root instance parameter configured.

SQL> show parameter wallet

NAME                                 TYPE                              VALUE
------------------------------------ --------------------------------- ------------------------------
ssl_wallet                           string
wallet_root                          string

And confirmed it was not in the pfile.

SQL> !grep -i wallet /u02/app/oracle/zdm/zdm_SRCSID_CHZ1_580/zdm/log/init_PDB001P1.ora.bck.before.restart

SQL>

We can easily find the directory where ZDM copied the wallet…

oracle@exacc-cl01n1:/u02/app/oracle/zdm/zdm_SRCSID_CHZ1_580/zdm/log/ [PDB001P1 (CDB$ROOT)] ls -ltrh /u02/app/oracle/zdm/zdm_SRCSID_CHZ1_580/zdm/TDE/tde
total 92K
-rw-r----- 1 oracle oinstall 2.5K Jun 27  2024 ewallet_2024062706422567.p12
-rw-r----- 1 oracle oinstall    0 Jun 27  2024 ewallet.p12.lck
-rw-r----- 1 oracle oinstall    0 Jun 27  2024 cwallet.sso.lck
-rw-r----- 1 oracle oinstall 4.0K Sep 19  2024 ewallet_2024091909561990_pre-refresh-AVQZZ_APP_001T.p12
-rw-r----- 1 oracle oinstall 5.2K Jan 16 10:54 ewallet_2025011609540081_pre-refresh-AVQZZ_APP_001T-20250116.p12
-rw-r----- 1 oracle oinstall 6.4K Feb 27 09:28 ewallet_2025022708284959_pre-refresh-AVQZZ_APP_001T-20250227.p12
-rw-r----- 1 oracle oinstall 7.7K Mar 31 14:10 ewallet_2025033112100549_pre-refresh-AVQZZ_APP_001T-20250331.p12
-rw-r----- 1 oracle oinstall 8.9K Jun 11 13:58 ewallet_2025061111581822_chg_password_to_prod.p12
-rw-r----- 1 oracle oinstall  11K Jun 11 14:00 ewallet.p12
-rw-r----- 1 oracle oinstall 8.9K Jun 11 14:00 ewallet_2025061112003957_pre-dryrun-SRCSID-to-prodAVAPCl.p12
-rw-r----- 1 oracle oinstall  11K Jun 11 14:00 cwallet.sso.back
-rw------- 1 oracle asmdba    11K Jun 12 10:03 cwallet.sso
oracle@exacc-cl01n1:/u02/app/oracle/zdm/zdm_SRCSID_CHZ1_580/zdm/log/ [PDB001P1 (CDB$ROOT)]

Let’s configure it!

oracle@exacc-cl01n1:/u02/app/oracle/zdm/zdm_SRCSID_CHZ1_580/zdm/log/ [PDB001P1 (CDB$ROOT)] sqh

SQL*Plus: Release 19.0.0.0.0 - Production on Thu Jun 12 12:34:45 2025
Version 19.26.0.0.0

Copyright (c) 1982, 2024, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.26.0.0.0

SQL> alter system set WALLET_ROOT='/u02/app/oracle/zdm/zdm_SRCSID_CHZ1_580/zdm/TDE' scope=spfile;

System altered.

Let’s restart the database again.

SQL> shutdown immediate
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.

SQL> startup
ORACLE instance started.

Total System Global Area 8577884400 bytes
Fixed Size                 13693168 bytes
Variable Size            3439329280 bytes
Database Buffers         5100273664 bytes
Redo Buffers               24588288 bytes
Database mounted.
ORA-28365: wallet is not open

Wallet is recognised but not opened.

Confirmed:

SQL> set lines 300
SQL> col WRL_PARAMETER for a30
SQL> col WRL_TYPE for a30
SQL> col WALLET_TYPE for a30
SQL> col status for a40
SQL> select WRL_PARAMETER, WRL_TYPE,WALLET_TYPE, status from V$ENCRYPTION_WALLET;

WRL_PARAMETER                  WRL_TYPE                       WALLET_TYPE                    STATUS
------------------------------ ------------------------------ ------------------------------ ----------------------------------------
                               FILE                           UNKNOWN                        NOT_AVAILABLE

tde_configuration parameter is not configured.

SQL> show parameter tde_configuration

NAME                                 TYPE                              VALUE
------------------------------------ --------------------------------- ------------------------------
tde_configuration                    string

Let’s configure it.

SQL> alter system set tde_configuration='keystore_configuration=FILE' scope=both;

System altered.

Checking, we can see that the wallet is now opened.

SQL> select WRL_PARAMETER, WRL_TYPE,WALLET_TYPE, status from V$ENCRYPTION_WALLET;

WRL_PARAMETER                    WRL_TYPE                       WALLET_TYPE                    STATUS
------------------------------   ------------------------------ ------------------------------ ----------------------------------------
/u02/app/oracle/zdm/zdm_SRCSID_C FILE                           AUTOLOGIN                      OPEN
HZ1_580/zdm/TDE/tde/

And we can open the database now.

SQL> alter database open;

Database altered.

SQL>

And completed the command for the TEMP tablespace encryption.

SQL> DROP TABLESPACE TEMP INCLUDING CONTENTS;

Tablespace dropped.

SQL> ALTER TABLESPACE TEMP_ENCRYPTED RENAME TO TEMP;

Tablespace altered.

SQL>

We can now resume the ZDM job. As the TEMP tablespace is now encrypted, ZDM will move forward with the next actions from the ZDM_POST_DATABASE_OPEN_TGT phase. Which is good because updating the ZDM XML metadata file to consider the phase as successful would have as consequence some missing steps and actions.

Resume ZDM job again:

[exauser@zdm-host ~]$ zdmcli resume job -jobid 580 -skip SWITCHOVER
zdm-host.domain.com: Audit ID: 6969

And we can see that the ZDM migration is now completed successfully.

[exauser@zdm-host ~]$ zdmcli query job -jobid 580
zdm-host.domain.com: Audit ID: 6970
Job ID: 580
User: exauser
Client: zdm-host
Job Type: "MIGRATE"
Scheduled job command: "zdmcli migrate database -sourcesid SRCSID -rsp /home/exauser/migration/zdm_SRCSID_physical_online.rsp -sourcenode dbsource-host -srcauth dbuser -srcarg1 user:oracle -srcarg2 identity_file:/home/exauser/.ssh/id_rsa -targetnode exacc-cl01n1 -tgtauth zdmauth -tgtarg1 user:opc -tgtarg2 identity_file:/home/exauser/.ssh/id_rsa -tgtarg3 sudo_location:/usr/bin/sudo -tdekeystorepasswd -tgttdekeystorepasswd -pauseafter ZDM_COPYFILES"
Scheduled job execution start time: 2025-06-11T17:09:40+02. Equivalent local time: 2025-06-11 17:09:40
Current status: SUCCEEDED
Result file path: "/u01/app/oracle/chkbase/scheduled/job-580-2025-06-11-17:10:06.log"
Metrics file path: "/u01/app/oracle/chkbase/scheduled/job-580-2025-06-11-17:10:06.json"
Job execution start time: 2025-06-11 17:10:07
Job execution end time: 2025-06-12 16:02:54
Job execution elapsed time: 4 hours 37 minutes 44 seconds
ZDM_GET_SRC_INFO .............. COMPLETED
ZDM_GET_TGT_INFO .............. COMPLETED
ZDM_PRECHECKS_SRC ............. COMPLETED
ZDM_PRECHECKS_TGT ............. COMPLETED
ZDM_SETUP_SRC ................. COMPLETED
ZDM_SETUP_TGT ................. COMPLETED
ZDM_PREUSERACTIONS ............ COMPLETED
ZDM_PREUSERACTIONS_TGT ........ COMPLETED
ZDM_VALIDATE_SRC .............. COMPLETED
ZDM_VALIDATE_TGT .............. COMPLETED
ZDM_DISCOVER_SRC .............. COMPLETED
ZDM_COPYFILES ................. COMPLETED
ZDM_PREPARE_TGT ............... COMPLETED
ZDM_SETUP_TDE_TGT ............. COMPLETED
ZDM_RESTORE_TGT ............... COMPLETED
ZDM_RECOVER_TGT ............... COMPLETED
ZDM_FINALIZE_TGT .............. COMPLETED
ZDM_CONFIGURE_DG_SRC .......... COMPLETED
ZDM_SWITCHOVER_SRC ............ COMPLETED
ZDM_SWITCHOVER_TGT ............ COMPLETED
ZDM_POST_DATABASE_OPEN_TGT .... COMPLETED
ZDM_DATAPATCH_TGT ............. COMPLETED
ZDM_NONCDBTOPDB_PRECHECK ...... COMPLETED
ZDM_NONCDBTOPDB_CONVERSION .... COMPLETED
ZDM_POST_MIGRATE_TGT .......... COMPLETED
ZDM_POSTUSERACTIONS ........... COMPLETED
ZDM_POSTUSERACTIONS_TGT ....... COMPLETED
ZDM_CLEANUP_SRC ............... COMPLETED
ZDM_CLEANUP_TGT ............... COMPLETED
[exauser@zdm-host ~]$

In the migrated PDB we can see that all tablespaces are encrypted!

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         3 PDB001P                 READ WRITE NO

SQL> select count(*) from  cdb_tablespaces a, (select x.con_id, y.ENCRYPTIONALG, x.name from V$TABLESPACE x,  V$ENCRYPTED_TABLESPACES y
  2    where x.ts#=y.ts# and x.con_id=y.con_id) b where a.con_id=b.con_id(+) and a.tablespace_name=b.name(+) and b.ENCRYPTIONALG='AES128';

  COUNT(*)
----------
      2803

SQL> select count(*) from  cdb_tablespaces a, (select x.con_id, y.ENCRYPTIONALG, x.name from V$TABLESPACE x,  V$ENCRYPTED_TABLESPACES y
  2    where x.ts#=y.ts# and x.con_id=y.con_id) b where a.con_id=b.con_id(+) and a.tablespace_name=b.name(+) and b.ENCRYPTIONALG is null;

  COUNT(*)
----------
         0

SQL> select a.con_id, a.tablespace_name, nvl(b.ENCRYPTIONALG,'NOT ENCRYPTED') from  cdb_tablespaces a, (select x.con_id, y.ENCRYPTIONALG, x.name from V$TABLESPACE x,  V$ENCRYPTED_TABLESPACES y
  2    where x.ts#=y.ts# and x.con_id=y.con_id) b where a.con_id=b.con_id(+) and a.tablespace_name=b.name(+) and b.ENCRYPTIONALG is null;

no rows selected

SQL>

To wrap up…

As explained in my previous blog, ZDM 21.5 version brings some correction, new functionnalities but also some new bug that I could easily mitigate.

L’article ZDM 21.5 Physical Online Migration failing during TEMP tablespace encryption est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/zdm-21-5-physical-online-migration-failing-during-temp-tablespace-encryption/feed/ 0 Oracle ZDM 21.5 new features https://www.dbi-services.com/blog/oracle-zdm-21-5-new-features/ https://www.dbi-services.com/blog/oracle-zdm-21-5-new-features/#respond Tue, 05 Aug 2025 09:10:13 +0000 https://www.dbi-services.com/blog/?p=39925 I have already written several articles about my experience migrating databases with Oracle Zero Downtime Migration. At that time, the version available was 21.4. The current version is now 21.5. Recently, I had the opportunity to patch our customer ZDM environment to 21.5 and to run new migrations in this new version. In this blog, […]

L’article Oracle ZDM 21.5 new features est apparu en premier sur dbi Blog.

]]> I have already written several articles about my experience migrating databases with Oracle Zero Downtime Migration. At that time, the version available was 21.4. The current version is now 21.5. Recently, I had the opportunity to patch our customer ZDM environment to 21.5 and to run new migrations in this new version. In this blog, I would like to share what new features brings the 21.5 version and also what I could see running the same kind of migration with the new version.

New 21.5 features for Physical Migration


Inflight Upgrades

We know that until 21.4, physical migration workflow required that source and target databases are in the same database major release. This of course due to Data Guard constrains.

With 21.5, Data Guard automation is changing to a migration methodology. There is no database release constraint any more. There is of course no change in the way Data Guard is working, but ZDM Physical Migration workflow now integrates and take in account in-flight upgrades. It is now possible to have source database in version 11.2.0.4 or 12c been migrated with physical online workflow to 19c or even from source 19c to 23ai.

In case source is a multitenant database, the temporary target database will be created in the same version as the source database. After switchover to this temporary database, ZDM will upgrade it using oracle tools.

In case source is a non-multitenant database, ZDM will perform the migration using a non-CDB temporary database and running a switchover to it, like I explained in my previous articles. Then ZDM will use autoupgrade to upgrade the database to the desired version and converting it to multitenant in the target CDB.

Cloud Native Disaster Recovery automation

Until 21.4, we had to manually create the Data Guard configuration in the cloud once the migration was performed. Version 21.5 now integrates a function to create Data Guard configuration in the target environment with ZDM once the migration is done. Customer can then benefit from a Cloud Native DR architecture having a standby database in the cloud.

New 21.5 features for Logical Migration


Oracle Autonomous Database as source

Oracle Autonomous Database as source database for migration is now supported. Migration from autonomous to autonomous is now possible. This will allow tiers, serverless and dedicated Exadata Infrastructure Autonomous Database move.

GoldenGate Enhancements

There is several enhancement for logical online migration supporting now following GoldenGate functions:

  • Integrated and non-integrated replication mode
  • Audit trail import
  • Large Transaction split
  • Pre-checks for ggadmin
  • Feature groups, Constraint Handling, Concurrent Migrations
  • User-specified GoldenGate Schema

Data Pump Enhancements

There is also further enhancements integrating following Data Pump functions:

  • Dump file retention and reuse
  • Advanced Queue Objects for post-import reload

Others…

There is some other new enhancements for logical workflow like:

  • OCI File Storage Servie (FSS) for data transfer medium with Oracle Autonomous Database as target
  • Automated refresh of Materialized Views if specified.

Offline Hybrid Migration

Version 21.5 brings a new migration method, the Offline Hybrid Migration using:

  • RMAN transportable tablespaces for Data Migration
  • Data Pump Export/Import for metadata

This method requires NFS as backup location and allows cross-endian and cross-version migration.

The supported targets are:

  • Oracle Base Database Services (BaseDB)
  • Oracle Exadata Database Service on Dedicated Infrastructure (ExaDB-D)
  • Oracle Exadata Database Service on Dedicated Infrastructure (ExaDB-D) on Oracle Database@Azure
  • Oracle Exadata database Service on Cloud@Customer (ExaDB-C@C)
  • Oracle Exadata On-premises

My testing feedback

Running the same migration method, I could see following enhancements from 21.4 to 21.5:

  • Bug correction like switchover issue (described in a previous blog)
  • TEMP, SYSTEM, SYSAUX and UNDO tablespaces are now automatically encrypted by ZDM. No need to do it manually after the migration.

On the other side, I could find a new bug during the TEMP tablespace encryption. The problem is that the temporary database needs to be restarted to complete deletion of the previous TEMP tablespace. I could easily resolve manually the problem and completed ZDM migration. I will describe this workaround in a further blog.

L’article Oracle ZDM 21.5 new features est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/oracle-zdm-21-5-new-features/feed/ 0 Automatisation for Oracle ZDM installation and update https://www.dbi-services.com/blog/automatisation-for-oracle-zdm-installation-and-update/ https://www.dbi-services.com/blog/automatisation-for-oracle-zdm-installation-and-update/#respond Thu, 31 Jul 2025 06:31:15 +0000 https://www.dbi-services.com/blog/?p=39773 Let’s have a look how we can install and update Oracle Zero Downtime Migration tool, and see how we can automatise it with ansible… Check ZDM build Let’s first see how we can check ZDM version using zdm cli. [zdm@zdmhost ~]$ zdmcli -v RHP_PT.ZDM21_LINUX.X64_221207.30 [zdm@zdmhost ~]$ zdmcli -build version: 21.0.0.0.0 full version: 21.4.0.0.0 patch version: […]

L’article Automatisation for Oracle ZDM installation and update est apparu en premier sur dbi Blog.

]]> Let’s have a look how we can install and update Oracle Zero Downtime Migration tool, and see how we can automatise it with ansible…

Check ZDM build

Let’s first see how we can check ZDM version using zdm cli.

[zdm@zdmhost ~]$ zdmcli -v
RHP_PT.ZDM21_LINUX.X64_221207.30

[zdm@zdmhost ~]$ zdmcli -build
version: 21.0.0.0.0
full version: 21.4.0.0.0
patch version: 21.4.5.0.0
label date: 221207.30
ZDM kit build date: Mar 21 2024 22:07:12 UTC
CPAT build version: 24.6.0
[exauser@sva-oelexa501 ~]$

Version installed is 21.4, and we would see how to update it to last current version, which is 21.5.

Manuel installation and update

Manuel installation and update is quite easy as it will be done through the zdminstall.sh script. It will be the option that will highlight if it is a first installation or an update. For a first installation, the option will be setup and for an update it will be update.

Automatisation of the installation and update of ZDM

The playbook to automatise the installation and the update, in my case named deploy_zdm.yml, is composed of following tasks.

We start by defining the playbook and the variable that will be used in the included tasks to install or update to zdm current last version, which is 21.5. The zip file to unarchive will be stored in the ../oracle_swfiles directory.

---
# Playbook for deploying ZDM host
# Marc Wagner - dbi
# Date : 06.10.2023
# 11.06.2025 : stop service + new 21.5 version
- name: Deploy ZDM
  hosts: "zdmhost"
  vars:
    sfw_folder: "../oracle_swfiles"


    zdm_base: "/u01/app/oracle"
    zdm_install_dir: "zdm21.5"
    zdm_download_dir: "/u01/app/zdm_download_dir"
    zdm_archive: "{{ sfw_folder }}/V1045330-01.zip"

    # ZDM update is an in-place process
    zdm_home: "/u01/app/oracle/product/zdm"
  environment:
    HTTP_PROXY: ""
    HTTPS_PROXY: ""

  tasks:

Then we will have all the tasks performing the installation.

The first one will be used to define the variable for the option provided as extra argument of the ansible playbook. And we will assert that the variable is provided.

- name: Assert extra-var has been passed as argument to playbook
  ansible.builtin.assert:
    that:
      - deploy_option is defined
    quiet: false
    fail_msg: "Please provide --extra-var deploy_option="
    success_msg: "deploy_option={{ deploy_option }}"

The next task will stop ZDM service and will only run if following file exists : /u01/app/oracle/product/zdm/bin/zdmservice.

This check is done so the task will only run if ZDM tool is already installed.

- name: Stop ZDM service
  ansible.builtin.shell: |
    cmd: |
    {{ zdm_home }}/bin/zdmservice stop
  args:
    executable: "/bin/bash"
    removes: "/u01/app/oracle/product/zdm/bin/zdmservice"

The next task will install some prerequisite if not already installed.

- name: Install ZDM software prerequisites
  become: true
  become_user: root
  ansible.builtin.dnf:
    name: "{{ item }}"
  loop:
    - perl
    - expect
    - libaio
    - glibc-devel
    - unzip
    - libnsl
    - ncurses-compat-libs
    - oraclelinux-developer-release-el8

The next task will create all needed directories like zdm base, zdm home and the file used to store the installation archive file.

- name: Create directories for ZDM tool
  become: true
  become_user: root
  ansible.builtin.file:
    path: "{{ item }}"
    state: directory
    owner: exauser
    group: exauser
    mode: '755'
  loop:
    - "{{ zdm_base }}"
    - "{{ zdm_home }}"
    - "{{ zdm_download_dir }}"

The next task will unarchive the installation zip file.

- name: Unarchive ZDM
  ansible.builtin.unarchive:
    src: "{{ zdm_archive }}"
    dest: "{{ zdm_download_dir }}"

And the next task will finally installed or update zdm tool according to the option given to the playbook.

- name: Install or update ZDM
  ansible.builtin.shell:
    cmd: |
      {{ zdm_download_dir }}/{{ zdm_install_dir }}/zdminstall.sh \
        {{ deploy_option }} oraclehome={{ zdm_home }} oraclebase={{ zdm_base }} \
        ziploc={{ zdm_download_dir }}/{{ zdm_install_dir }}/zdm_home.zip -zdm
  args:
    executable: "/bin/bash"

And we can finally with the last steps start ZDM service.

- name: Start ZDM service
  ansible.builtin.shell: |
    cmd: |
    {{ zdm_home }}/bin/zdmservice start
  args:
    executable: "/bin/bash"

Run the playbook

As prerequisite, let’s first check that the appropriate ZDM installation zip file is in the expected ansible folder.

[myuser@domain.com@admin-host zdm_ansible]$ ls -ltrh ./oracle_swfiles/
total 874M
-rw-r--r--. 1 myuser@domain.com myuser@domain.com 871M Jun 11 10:12 V1045330-01.zip
[myuser@domain.com@admin-host zdm_ansible]$

We can check that the assert task to ensure we put the appropriate –extra-vars works.

[myuser@domain.com@admin-host zdm_ansible]$ ansible-playbook ./playbooks/deploy_zdm.yml

PLAY [Deploy ZDM] ***********************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************************************************************************************
[WARNING]: Platform linux on host zdmhost is using the discovered Python interpreter at /usr/bin/python3, but future installation of another Python interpreter could change the meaning of that path. See https://docs.ansible.com/ansible-
core/2.15/reference_appendices/interpreter_discovery.html for more information.
ok: [zdmhost]

TASK [Assert extra-var has been passed as argument to playbook] *************************************************************************************************************************************************************************************************************************
fatal: [zdmhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'deploy_option' is undefined. 'deploy_option' is undefined\n\nThe error appears to be in '/home/nfs/domain.com/myuser/ExaCCGit/zdm_ansible/playbooks/deploy_zdm.yml': line 25, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n    - name: Assert extra-var has been passed as argument to playbook\n      ^ here\n"}

PLAY RECAP ******************************************************************************************************************************************************************************************************************************************************************************
zdmhost              : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

[myuser@domain.com@admin-host zdm_ansible]$

Then we can run the playbook with the update option.

[myuser@domain.com@admin-host zdm_ansible]$ ansible-playbook ./playbooks/deploy_zdm.yml -e deploy_option="update"

PLAY [Deploy ZDM] ***********************************************************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************************************************************************************
[WARNING]: Platform linux on host zdmhost is using the discovered Python interpreter at /usr/bin/python3, but future installation of another Python interpreter could change the meaning of that path. See https://docs.ansible.com/ansible-
core/2.15/reference_appendices/interpreter_discovery.html for more information.
ok: [zdmhost]

TASK [Assert extra-var has been passed as argument to playbook] *************************************************************************************************************************************************************************************************************************
ok: [zdmhost] => {
    "changed": false,
    "msg": "deploy_option=update"
}

TASK [Stop ZDM service] *****************************************************************************************************************************************************************************************************************************************************************
changed: [zdmhost]

TASK [Install ZDM software prerequisites] ***********************************************************************************************************************************************************************************************************************************************
ok: [zdmhost] => (item=perl)
ok: [zdmhost] => (item=expect)
ok: [zdmhost] => (item=libaio)
ok: [zdmhost] => (item=glibc-devel)
ok: [zdmhost] => (item=unzip)
ok: [zdmhost] => (item=libnsl)
ok: [zdmhost] => (item=ncurses-compat-libs)
ok: [zdmhost] => (item=oraclelinux-developer-release-el8)

TASK [Create directories for ZDM tool] **************************************************************************************************************************************************************************************************************************************************
ok: [zdmhost] => (item=/u01/app/oracle)
ok: [zdmhost] => (item=/u01/app/oracle/product/zdm)
ok: [zdmhost] => (item=/u01/app/zdm_download_dir)

TASK [Unarchive ZDM] ********************************************************************************************************************************************************************************************************************************************************************
ok: [zdmhost]

TASK [Install or update ZDM] ************************************************************************************************************************************************************************************************************************************************************
changed: [zdmhost]

TASK [Start ZDM service] ****************************************************************************************************************************************************************************************************************************************************************
changed: [zdmhost]

PLAY RECAP ******************************************************************************************************************************************************************************************************************************************************************************
zdmhost              : ok=8    changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Check new ZDM version

And we can check that the new ZDM tool version is 21.5.

[zdm@zdmhost ~]$ zdmcli -v
RHP_PT.ZDM21_LINUX.X64_240219.12

[zdm@zdmhost ~]$ zdmcli -build
version: 21.0.0.0.0
full version: 21.5.0.0.0
patch version: N/A
label date: 240219.12
ZDM kit build date: Sep 10 2024 21:59:18 UTC
CPAT build version: 24.6.0
[zdm@zdmhost ~]$

To wrap up…

Installing and updating ZDM cli is quite easy, and writing an ansible playbook will help automatising the installation and further update.

L’article Automatisation for Oracle ZDM installation and update est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/automatisation-for-oracle-zdm-installation-and-update/feed/ 0