Physical Online Migration to ExaCC with Oracle Zero Downtime Migration (ZDM)

A while ago I had been testing and blogging about ZDM, see my previous articles. And I finally had the chance to implement it at one of our customer to migrate on-premise databases to Exadata Cloud @Customer. After having been implementing Logical Offline migration with ZDM, see my previous article, https://www.dbi-services.com/blog/logical-offline-migration-to-exacc-with-oracle-zero-downtime-migration-zdm/, I had the opportunity to implement a Physical Online Migration and testing it with one database taken as pilot. In this article I would like to share with you my experience I could get from migrating an on-premises database to ExaCC using ZDM Physical Online Migration. This method will use Data Guard and we can then only use it to migrate Oracle Enterprise Edition databases. We call it Physical Online, because ZDM will create a Standby database either through a backup or with Direct Data Transfer (active duplication with rman or restore from service) and synchronise it with the primary. During all the preparation the database is still available for the application, and the maintenance windows will be shorter with less downtime. It will just be needed for the switchover operation. Of course ZDM can include non-cdb to pdb conversion which will make it a little bit longer. The Physical Online is the only ZDM method including fallback. We intended to use this method at customer side, as mandatory one for Large Oracle EE databases, and preferred one for Small Oracle EE databases.

Introduction

The on-premise databases are single-tenant (non-cdb) database running for all version 19.10.

The target databases are Oracle RAC databases running on ExaCC with Oracle version 19.21.

The Oracle Net port used on the on-premise site is 13000 and the Oracle Net port used on the ExaCC is 1521.

We will use ZDM to migrate the on-premise single-tenant database, to a PDB within a CDB. ZDM will then be in charge of migrating the database to the exacc using Data Guard, run datapatch, convert non-cdb database to pdb within a target cdb, upgrade Time Zone. The creation of the standby database will be done through a direct connection. Without any backup.

Of course I have anonymised all outputs to remove customer infrastructure names. So let’s take following convention.

ExaCC Cluster 01 node 01 : ExaCC-cl01n1
ExaCC Cluster 01 node 02 : ExaCC-cl01n2
On premises Source Host : vmonpr
Target db_unique_name on the ExaCC : ONPR_RZ2
Database Name to migrate : ONPR
ZDM Host : zdmhost
ZDM user : zdmuser
Domain : domain.com
ExaCC PDB to migrate to : ONPRZ_APP_001T

We will then migrate on-premise Single-Tenant database, named ONPR, to a PDB on the ExaCC. The PDB will be named ONPRZ_APP_001T.

Ports

It is important to mention that following ports are needed:

If Oracle Net ports are for example not opened from the Exacc to the on-premise host, the migration evaluation will immediately stopped at one of the first steps named ZDM_PRECHECKS_TGT, and following errors will be found in the log file:

PRGZ-1132 : -eval failed for the phase ZDM_PRECHECKS_TGT with exception
PRGZ-3176 : a database connection cannot be established from target node ExaCC-cl01n1 to source node vmonpr
PRCC-1021 : One or more of the submitted commands did not execute successfully.
PRCZ-2103 : Failed to execute command "/u02/app/oracle/product/19.0.0.0/dbhome_2/bin/tnsping" on node "ExaCC-cl01n1" as user "root". Detailed error:
TNS Ping Utility for Linux: Version 19.0.0.0.0 - Production on 06-FEB-2024 10:06:37

Copyright (c) 1997, 2023, Oracle.  All rights reserved.

Used parameter files:

Used HOSTNAME adapter to resolve the alias
Attempting to contact (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=tcp)(HOST=X.X.X.15)(PORT=13000)))
TNS-12535: TNS:operation timed out
PRCC-1025 : Command submitted on node ExaCC-cl01n2 timed out after 60 seconds..

If Oracle Net ports are for example not opened from the on-premise host to ExaCC, the migration evaluation will immediately stopped at one of the other steps named ZDM_PRECHECKS_SRC, and following errors will be found in the log file:

PRGZ-1132 : -eval failed for the phase ZDM_PRECHECKS_SRC with exception
PRGZ-3130 : failed to establish connection to target listener from nodes [vmonpr]
PRCC-1021 : One or more of the submitted commands did not execute successfully.
PRCC-1025 : Command submitted on node vmonpr timed out after 15 seconds...

Requirements… To be known before starting

There are few requirements that are needed.

ZDM Host

• SSH connection allowed between Source and Target host
• SSH authentication key pairs without passphrase should be established and tested for the user between ZDM host and both source and target database

Source Database

• Both source and target need to be on the same release version.
• Transparent Data Encryption (TDE) wallet must be configured (even if source database is not encrypted)
• WALLET_TYPE should be set to either AUTOLOGIN or PASSWORD
• Wallet STATUS should be OPEN
• Wallet should be opened on all PDB in case the source is a container Database
• The master key must be set for all the PDB and the container database
• In case the source database is a RAC database, SNAPSHOT CONTROLFILE must be configured on a shared location on all cluster nodes
• SCAN listener/listener connections allowed on both source and target DB
• DB_UNIQUE_NAME parameter must be different than target database
• SYSPASSWORD must be the same on the source and target database

Target Database

• Database must be created prior the migration
• Database release version should match source version.
• The target database patch level should also be the same or higher than the source database. In case the target database patch level is higher, ZDM can be configured to run datapatch on the target database. Target database patch level can not be lower than source database.
• For Oracle RAC databases, SSH connectivity between nodes for the oracle user should be setup
• Storage size should be sufficient (same as source database)
• DB_NAME parameter must be the same than the source database
• DB_UNIQUE_NAME parameter must be different than the one on the source database
• Automatic backups should be disabled (for ExaC@C section configure backups, option backup destination, none should be selected)
• TDE should be activated
• Wallet should be open and WALLET_TYPE should be set to either AUTOLOGIN or PASSWORD
• SYSPASSWORD must be the same on the source and target database

Others

• Ensure that all ports have been opened.
• Oracle NET Services should be configured and tested on both source and target database for Data Guard synchronisation and deployment of the standby database with active duplication
• We will need to go through a temporary multitenant database on the ExaCC which will have same DB_NAME than the source and different DB_UNIQUE_NAME. This CDB will host the final PDB.
• The final PDB can then be relocated to the appropriate final CDB on the ExaCC laster on.
• ZDM will create its own temporary database with DB_NAME as source database and DB_UNIQUE_NAME as final PDB name to build the Data Guard and will remove it during cleanup phase

Prepare ZDM Physical Online Response file

We will prepare the ZDM response file that will be used, by copying the template provided by ZDM:

[zdmuser@zdmhost migration]$ cp -p /u01/app/oracle/product/zdm/rhp/zdm/template/zdm_template.rsp ./zdm_ONPR_physical_online.rsp

The main parameters to take care of are :

Note that there is no parameter for the listener port on the target (ExaCC) so assuming this is hard coded to use default 1521 port.

Also note that as we configured SHUTDOWN_SRC as FALSE, additionnal steps will be required to ensure that application do not use the SOURCE (on-premise) database any more.

Updated ZDM response file compared to ZDM template for the migration we are going to run:

[zdmuser@zdmhost migration]$ diff zdm_ONPR_physical_online.rsp /u01/app/oracle/product/zdm/rhp/zdm/template/zdm_template.rsp
24c24
< TGT_DB_UNIQUE_NAME=ONPR_RZ2
---
> TGT_DB_UNIQUE_NAME=
32c32
< MIGRATION_METHOD=ONLINE_PHYSICAL
---
> MIGRATION_METHOD=
63c63
< DATA_TRANSFER_MEDIUM=DIRECT
---
> DATA_TRANSFER_MEDIUM=
75c75
< PLATFORM_TYPE=EXACC
---
> PLATFORM_TYPE=
119c119
< SRC_DB_LISTENER_PORT=13000
---
> SRC_DB_LISTENER_PORT=
230c230
< NONCDBTOPDB_CONVERSION=TRUE
---
> NONCDBTOPDB_CONVERSION=FALSE
252c252
< SKIP_FALLBACK=FALSE
---
> SKIP_FALLBACK=
268c268
< TGT_RETAIN_DB_UNIQUE_NAME=TRUE
---
> TGT_RETAIN_DB_UNIQUE_NAME=
312c312
< SHUTDOWN_SRC=FALSE
---
> SHUTDOWN_SRC=
333c333
< SRC_RMAN_CHANNELS=3
---
> SRC_RMAN_CHANNELS=
340c340
< TGT_RMAN_CHANNELS=6
---
> TGT_RMAN_CHANNELS=
526c526
< ZDM_USE_DG_BROKER=TRUE
---
> ZDM_USE_DG_BROKER=
574c574
< ZDM_NONCDBTOPDB_PDB_NAME=ONPRZ_APP_001T
---
> ZDM_NONCDBTOPDB_PDB_NAME=
595c595
< ZDM_TGT_UPGRADE_TIMEZONE=TRUE
---
> ZDM_TGT_UPGRADE_TIMEZONE=FALSE

ZDM Build Version

We are using ZDM build version 21.4:

[zdmuser@zdmhost migration]$ /u01/app/oracle/product/zdm/bin/zdmcli -build
version: 21.0.0.0.0
full version: 21.4.0.0.0
patch version: 21.4.1.0.0
label date: 221207.25
ZDM kit build date: Jul 31 2023 14:24:25 UTC
CPAT build version: 23.7.0

Passwordless Login

Passwordless Login needs to be configured between ZDM Host, the Source Host and Target Host. See my previous blog : https://www.dbi-services.com/blog/oracle-zdm-migration-java-security-invalidkeyexception-invalid-key-format/

If Passwordless Login is not configured with one node, you will see such error in the log file during migration evaluation:

PRCZ-2006 : Unable to establish SSH connection to node "ExaCC-cl01n2" to execute command "/u02/app/oracle/product/19.0.0.0/dbhome_2/bin/tnsping vmonpr:13000"

Creation of the target database

As explained in the requirements, we must create a target CDB on the ExaCC with same DB_NAME as the source database to be migrated but other DB_UNIQUE_NAME. In our case it will be ONPR for the DB_NAME and ONPR_RZ2 for the DB_UNIQUE_NAME. This database must exist before the migration is started with ZDM. ZDM will create another temporary database taking the final PDB name and will use this target CDB as a template.

TDE (Transparent Data Encryption) configuration

The source database doesn’t need to be encrypted. The target database will be encrypted in any case. ZDM supports the migration of an encrypted and non-encrypted source database. The target database encryption will be taken in account during migration process. Even if the source database is not encrypted a TDE wallet still needs to be configured prior the migration as ZDM will use it to encrypt data to the target.

We need to note that a downtime is needed to reboot the database when wallet_root parameter needs to be configured.

Also until the migration is completed it is more than recommended that the wallet is part of the backup strategy.

Configure instance parameter

Check that the WALLET directory exits otherwise create it:

SQL> !ls /u00/app/oracle/admin/ONPR/wallet
ls: cannot access /u00/app/oracle/admin/ONPR/wallet: No such file or directory

SQL> !mkdir /u00/app/oracle/admin/ONPR/wallet

Configure instance parameter for the database wallet and restart the database:

SQL> alter system set WALLET_ROOT='/u00/app/oracle/admin/ONPR/wallet' scope=spfile;

SQL> shutdown immediate

SQL> startup

Check the wallet. No WRL_PARAMETER should be displayed. WALLET_TYPE should be unknown and STATUS not_available.

SQL> select WRL_PARAMETER, WRL_TYPE,WALLET_TYPE, status from V$ENCRYPTION_WALLET;

WRL_PARAMETER                  WRL_TYPE             WALLET_TYPE          STATUS
------------------------------ -------------------- -------------------- ---------------------------
                               FILE                 UNKNOWN              NOT_AVAILABLE

Configure TDE:

SQL> alter system set tde_configuration='keystore_configuration=FILE' scope=both;

System altered.

Check the Wallet. WRL_PARAMETER should be displayed with the wallet location. WALLET_TYPE should still be unknown and STATUS not_available.

SQL> select WRL_PARAMETER, WRL_TYPE,WALLET_TYPE, status from V$ENCRYPTION_WALLET;

WRL_PARAMETER                            WRL_TYPE             WALLET_TYPE          STATUS
---------------------------------------- -------------------- -------------------- -----------------
/u00/app/oracle/admin/ONPR/wallet/tde/   FILE                 UNKNOWN              NOT_AVAILABLE

Create keystore

Create the keystore using appropriate ExaCC password. We recommend to use the same one for source and target, albeit they can be different.

SQL> ADMINISTER KEY MANAGEMENT CREATE KEYSTORE IDENTIFIED BY "*********************"

keystore altered.

We now can see that we have a wallet file in the TDE wallet directory:

SQL> !ls -ltrh /u00/app/oracle/admin/ONPR/wallet/tde/
total 4.0K
-rw-------. 1 oracle dba 2.5K Feb  9 16:11 ewallet.p12

And if we check the wallet status, we can see it is still UNKNOWN for the WALLET_TYPE, but now STATUS is set to CLOSED.

SQL> select WRL_PARAMETER, WRL_TYPE,WALLET_TYPE, status from V$ENCRYPTION_WALLET;

WRL_PARAMETER                            WRL_TYPE             WALLET_TYPE          STATUS
---------------------------------------- -------------------- -------------------- ------------------------------
/u00/app/oracle/admin/ONPR/wallet/tde/   FILE                 UNKNOWN              CLOSED

Open the keystore

The keystore can now be opened.

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "***************";

keystore altered.

And the wallet type is now set to PASSWORD and status is OPEN_NO_MASTER_KEY.

SQL> select WRL_PARAMETER, WRL_TYPE,WALLET_TYPE, status from V$ENCRYPTION_WALLET;

WRL_PARAMETER                            WRL_TYPE             WALLET_TYPE          STATUS
---------------------------------------- -------------------- -------------------- ------------------------------
/u00/app/oracle/admin/ONPR/wallet/tde/   FILE                 PASSWORD             OPEN_NO_MASTER_KEY

Create and activate the master encryption key

Using same password we can now create and activate the master encryption key using backup option. This will set the database encryption key into the wallet.

SQL> ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY "*************" with backup;

keystore altered.

If you are running version 19.10, as we are here, you will face following error:

ORA-28374: typed master key not found in wallet

This is related to following bug:

Bug 31500699 – ORA-28374: typed master key not found in wallet after tablespace TDE Enabled ( Doc ID 31500699.8 )

This is not an issue, we can move forward as the master encryption key has been anyhow created and added in the wallet. The only problem would be that it will impossible to encrypt any data. We do not care as we are not encrypting the source on-premise database and we should not, as we are not licensed with Oracle Advanced Security.

We now have a new wallet and a backup one:

SQL> !ls -ltrh /u00/app/oracle/admin/ONPR/wallet/tde/
total 8.0K
-rw-------. 1 oracle dba 2.5K Feb  9 16:16 ewallet_2024020915161059.p12
-rw-------. 1 oracle dba 4.0K Feb  9 16:16 ewallet.p12

Set autologin Wallet

We will change the wallet type from password to autologin using the same password, in order for the wallet to be opened automatically.

SQL> ADMINISTER KEY MANAGEMENT CREATE AUTO_LOGIN KEYSTORE FROM KEYSTORE '/u00/app/oracle/admin/ONPR/wallet/tde/' IDENTIFIED BY "************";

keystore altered.

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE CLOSE IDENTIFIED BY "******************";

keystore altered.

And we can check that all has been configured appropriately for the wallet:

SQL> select WRL_PARAMETER, WRL_TYPE,WALLET_TYPE, status from V$ENCRYPTION_WALLET;

WRL_PARAMETER                            WRL_TYPE             WALLET_TYPE          STATUS
---------------------------------------- -------------------- -------------------- ------------------------------
/u00/app/oracle/admin/ONPR/wallet/tde/   FILE                 AUTOLOGIN            OPEN

And we can see that we have now an autologin cwallet.sso file:

SQL> !ls -ltrh /u00/app/oracle/admin/ONPR/wallet/tde/
total 14K
-rw-------. 1 oracle dba 4.0K Feb  9 17:07 ewallet.p12
-rw-------. 1 oracle dba 5.7K Feb  9 17:07 cwallet.sso
-rw-------. 1 oracle dba 2.5K Feb  9 16:16 ewallet_2024020915161059.p12

Change tablespace_encryption instance parameter to decrypt_only

If we do not have any Oracle Advanced Security License on our on-premise environment, it is really mandatory to make sure not to encrypt any new tablespace created on the source (on-premise) once TDE is configured, otherwise Oracle Advanced Security License will be activated. It is also mandatory to decrypt the encrypted redo coming from the ExaCC and applied on the on-premise database. TABLESPACE_ENCRYPTION parameter is an alternative to the ENCRYPT_NEW_TABLESPACES parameter. After ZDM switchover steps will be completed, the primary database will be running on the ExaCC, and tablespaces will be encrypted on the ExaCC. The generated redo logs that need to be applied on the standby database running on the on-premise environment will be encrypted as well. If we are not licensed with Oracle Advanced Security on the on-premise environment, we also need to have this TABLESPACE_ENCRYPTION parameter set to decrypt_only to be able to decrypt the redo before applying them.

ENCRYPT_NEW_TABLESPACES is also an important parameter:

• If set to CLOUD_ONLY and new tablespace is created on the Oracle Cloud, the tablespace is transparently encrypted even if the ENCRYPTION … ENCRYPT clause for the SQL CREATE TABLESPACE statement is not specified.
• If set to CLOUD_ONLY and new tablespace is created in an on-premises database, the tablespace is only encrypted if especially mentioned in the ENCRYPTION … ENCRYPT clause for the SQL CREATE TABLESPACE statement.
• If set to ALWAYS, either we are in the Oracle Cloud or on on-premises, the new tablespace will be transparently encrypted, even if the ENCRYPTION … ENCRYPT clause for the SQL CREATE TABLESPACE statement is not specified. As we do not have any Oracle Advanced Security Licences on our on-premise, this value should never be set on the on-premise databases.
• If set to DDL, either we are in the Oracle Cloud or on on-premise, the new tablespace create statement will tell if the tablespace will be encrypted or not. If no ENCRYPTION clause is specified, then the new tablespace will not be encrypted. The new tablespace will only be encrypted if especially mentioned in the ENCRYPTION … ENCRYPT clause for the SQL CREATE TABLESPACE statement.

See Oracle documentation:

Oracle documentation for TABLESPACE_ENCRYPTION

Oracle documentation for ENCRYPT_NEW_TABLESPACES

In our case, knowing we do not have any Oracle Advanced Security License for our on-premise, we will ensure to set the TABLESPACE_ENCRYPTION instance parameter to DECRYPT_ONLY value and ensure to set the ENCRYPT_NEW_TABLESPACES on either CLOUD_ONLY or DDL value.

Unfortunately the parameter tablespace_encryption came only with Oracle 19.16 version. So if you are running an older version on the source on-premise database, you do not have the possibility to use this parameter.

This means that you will need to:

• Deactivate fallback possibility. You will only be able, in this situation, to use ZDM to migrate the database but without any fallback possibilities. Bad situation…
• You will need to ensure the parameter ENCRYPT_NEW_TABLESPACES is set to DDL and ensure no ENCRYPTION clause is specified in the statement for any new created tablespace

SQL> show parameter encry

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
encrypt_new_tablespaces              string      CLOUD_ONLY
tablespace_encryption                string      MANUAL_ENABLE

SQL> alter system set tablespace_encryption='decrypt_only' scope=spfile;

System altered.

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> startup
ORACLE instance started.

Total System Global Area 3.7581E+10 bytes
Fixed Size                 23061704 bytes
Variable Size            5100273664 bytes
Database Buffers         3.2346E+10 bytes
Redo Buffers              111153152 bytes
Database mounted.
Database opened.

SQL> show parameter encry

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
encrypt_new_tablespaces              string      DDL
tablespace_encryption                string      DECRYPT_ONLY

Update SYS user password on the on-premise database

Both source and target SYS users password should match.

Update the source one with the same ExaCC one you are using:

SQL> alter user sys identified by "********";

Update source listener.ora with static entry

If the on-premise source database is not running on oracle restart (grid infra), you will have to add a static entry for DGMGRL service on the appropriate listener. Unfortunately ZDM does not do it.

SID_LIST_<listener_name> =
  (SID_LIST =
    (SID_DESC = 
      (GLOBAL_DBNAME = <dbname> _DGMGRL.<domain>) 
      (ORACLE_HOME = <ORACLE_HOME>) 
      (SID_NAME = <SID>)
    )
  )

If you do not do so and you will resume the migration, you will have the ZDM switchover steps, ZDM_SWITCHOVER_SRC, failing with following error:

PRGZ-3605 : Oracle Data Guard Broker switchover to database "ONPRZ_APP_001T" on database "ONPR" failed.
ONPRZ_APP_001T
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Thu Feb 22 09:46:57 2024
Version 19.10.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected to "ONPR"
Connected as SYSDG.
DGMGRL> Performing switchover NOW, please wait...
Operation requires a connection to database "onprz_app_001t"
Connecting ...
Connected to "ONPRZ_APP_001T"
Connected as SYSDBA.
New primary database "onprz_app_001t" is opening...
Operation requires start up of instance "ONPR" on database "onpr"
Starting instance "ONPR"...
Unable to connect to database using (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(Host=vmONPR.domain.com)(Port=13000))(CONNECT_DATA=(SERVICE_NAME=ONPR_DGMGRL.domain.com)(INSTANCE_NAME=ONPR)(SERVER=DEDICATED)))
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Archived log backup on the source

You will have to ensure that the source database archived log deletion policy is set appropriately and ensure not to remove any archived log that would not be applied on the standby. This all to ensure no source archived log is missing for Data Guard.

Convert target database to single instance

I have converted the target database on the ExaCC used during ZDM migration (the one taken as template by ZDM and where the final PDB will be hosted), from RAC to single instance. And this for 2 reasons:

• The first one, as we will see later, ZDM will create the standby database on a new instance, using PDB final name as ORACLE_SID. And this temporary database is any how single instance
• If the target database is RAC, ZDM will create a second UNDO tablespace in the single instance source database. I do not want to make any change in the source database. Also as I’m running version 19.10 on the source, the UNDO will be encrypted and more over I will face bug 31500699 and ZDM migration will fail in error.

Update cluster_database instance parameter

oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] sqh

SQL> show parameter cluster_database

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
cluster_database                     boolean     TRUE
cluster_database_instances           integer     2

SQL> set line 300
SQL> col name for a30
SQL> col value for a30
SQL> select inst_id, name, value from gv$parameter where lower(name)='cluster_database';

   INST_ID NAME                           VALUE
---------- ------------------------------ ------------------------------
         2 cluster_database               TRUE
         1 cluster_database               TRUE

SQL> alter system set cluster_database=FALSE scope=spfile sid='*';

System altered.

Stop cluster database

oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] srvctl status database -d ONPR_RZ2
Instance ONPR1 is running on node ExaCC-cl01n1
Instance ONPR2 is running on node ExaCC-cl01n2

oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] srvctl stop database -d ONPR_RZ2

oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] srvctl status database -d ONPR_RZ2
Instance ONPR1 is not running on node ExaCC-cl01n1
Instance ONPR2 is not running on node ExaCC-cl01n2

Change grid infrastructure configuration

We will remove second instance.

oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] srvctl config database -d ONPR_RZ2
Database unique name: ONPR_RZ2
Database name: ONPR
Oracle home: /u02/app/oracle/product/19.0.0.0/dbhome_2
Oracle user: oracle
Spfile: +DATAC1/ONPR_RZ2/PARAMETERFILE/spfile.634.1160214211
Password file: +DATAC1/ONPR_RZ2/PASSWORD/pwdonpr_rz2.562.1160213439
Domain: domain.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATAC1
Mount point paths: /acfs01
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: racoper
Database instances: ONPR1,ONPR2
Configured nodes: ExaCC-cl01n1,ExaCC-cl01n2
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed
oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)]

oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] srvctl remove instance -d ONPR_RZ2 -i ONPR2
Remove instance from the database ONPR_RZ2? (y/[n]) y
oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)]

oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] srvctl config database -d ONPR_RZ2
Database unique name: ONPR_RZ2
Database name: ONPR
Oracle home: /u02/app/oracle/product/19.0.0.0/dbhome_2
Oracle user: oracle
Spfile: +DATAC1/ONPR_RZ2/PARAMETERFILE/spfile.634.1160214211
Password file: +DATAC1/ONPR_RZ2/PASSWORD/pwdonpr_rz2.562.1160213439
Domain: domain.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATAC1
Mount point paths: /acfs01
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: racoper
Database instances: ONPR1
Configured nodes: ExaCC-cl01n1
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed

Start target database ONPR on ExaCC

oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] srvctl start database -d ONPR_RZ2

oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] srvctl status database -d ONPR_RZ2
Instance ONPR1 is running on node ExaCC-cl01n1

As we can see, only one instance is running. This can also be double checked with the instance parameter.

oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] sqh

SQL> set lines 300 pages 500
SQL> col name for a30
SQL> col value for a30
SQL> select inst_id, name, value from gv$parameter where lower(name)='cluster_database';

   INST_ID NAME                           VALUE
---------- ------------------------------ ------------------------------
         1 cluster_database               FALSE

Evaluating ZDM Migration

We are now all ready to evaluate ZDM Migration. We will first run zdmcli with the -eval option to evaluate the migration and test if all is ok.

We need to provide some arguments :

All steps done for evaluation have been completed successfully:

[zdmuser@zdmhost migration]$ /u01/app/oracle/product/zdm/bin/zdmcli migrate database -sourcesid ONPR -rsp /home/zdmuser/migration/zdm_ONPR_physical_online.rsp -sourcenode vmonpr -srcauth zdmauth -srcarg1 user:oracle -srcarg2 identity_file:/home/zdmuser/.ssh/id_rsa -srcarg3 sudo_location:/usr/bin/sudo -targetnode ExaCC-cl01n1 -tgtauth zdmauth -tgtarg1 user:opc -tgtarg2 identity_file:/home/zdmuser/.ssh/id_rsa -tgtarg3 sudo_location:/usr/bin/sudo -tdekeystorepasswd -tgttdekeystorepasswd -eval
zdmhost.domain.com: Audit ID: 428
Enter source database ONPR SYS password:
Enter source database ONPR TDE keystore password:
Enter target container database TDE keystore password:
zdmhost: 2024-02-14T13:18:19.773Z : Processing response file ...
Operation "zdmcli migrate database" scheduled with the job ID "39".

[zdmuser@zdmhost migration]$ /u01/app/oracle/product/zdm/bin/zdmcli query job -jobid 39
zdmhost.domain.com: Audit ID: 434
Job ID: 39
User: zdmuser
Client: zdmhost
Job Type: "EVAL"
Scheduled job command: "zdmcli migrate database -sourcesid ONPR -rsp /home/zdmuser/migration/zdm_ONPR_physical_online.rsp -sourcenode vmonpr -srcauth zdmauth -srcarg1 user:oracle -srcarg2 identity_file:/home/zdmuser/.ssh/id_rsa -srcarg3 sudo_location:/usr/bin/sudo -targetnode ExaCC-cl01n1 -tgtauth zdmauth -tgtarg1 user:opc -tgtarg2 identity_file:/home/zdmuser/.ssh/id_rsa -tgtarg3 sudo_location:/usr/bin/sudo -tdekeystorepasswd -tgttdekeystorepasswd -eval"
Scheduled job execution start time: 2024-02-14T14:18:19+01. Equivalent local time: 2024-02-14 14:18:19
Current status: SUCCEEDED
Result file path: "/u01/app/oracle/chkbase/scheduled/job-39-2024-02-14-14:18:29.log"
Metrics file path: "/u01/app/oracle/chkbase/scheduled/job-39-2024-02-14-14:18:29.json"
Job execution start time: 2024-02-14 14:18:29
Job execution end time: 2024-02-14 14:21:18
Job execution elapsed time: 2 minutes 48 seconds
ZDM_GET_SRC_INFO ........... PRECHECK_PASSED
ZDM_GET_TGT_INFO ........... PRECHECK_PASSED
ZDM_PRECHECKS_SRC .......... PRECHECK_PASSED
ZDM_PRECHECKS_TGT .......... PRECHECK_PASSED
ZDM_SETUP_SRC .............. PRECHECK_PASSED
ZDM_SETUP_TGT .............. PRECHECK_PASSED
ZDM_PREUSERACTIONS ......... PRECHECK_PASSED
ZDM_PREUSERACTIONS_TGT ..... PRECHECK_PASSED
ZDM_VALIDATE_SRC ........... PRECHECK_PASSED
ZDM_VALIDATE_TGT ........... PRECHECK_PASSED
ZDM_POSTUSERACTIONS ........ PRECHECK_PASSED
ZDM_POSTUSERACTIONS_TGT .... PRECHECK_PASSED
ZDM_CLEANUP_SRC ............ PRECHECK_PASSED
ZDM_CLEANUP_TGT ............ PRECHECK_PASSED

Run the Migration with ZDM

We will run the migration adding a pause after the ZDM steps ZDM_CONFIGURE_DG_SRC. So ZDM will prepare all the environment (setting the environment, creating standby and configuring Data Guard). All this steps can be done without any downtime.

[zdmuser@zdmhost migration]$ /u01/app/oracle/product/zdm/bin/zdmcli migrate database -sourcesid ONPR -rsp /home/zdmuser/migration/zdm_ONPR_physical_online.rsp -sourcenode vmonpr -srcauth zdmauth -srcarg1 user:oracle -srcarg2 identity_file:/home/zdmuser/.ssh/id_rsa -srcarg3 sudo_location:/usr/bin/sudo -targetnode ExaCC-cl01n1 -tgtauth zdmauth -tgtarg1 user:opc -tgtarg2 identity_file:/home/zdmuser/.ssh/id_rsa -tgtarg3 sudo_location:/usr/bin/sudo -tdekeystorepasswd -tgttdekeystorepasswd -pauseafter ZDM_CONFIGURE_DG_SRC
zdmhost.domain.com: Audit ID: 543
Enter source database ONPR SYS password:
Enter source database ONPR TDE keystore password:
Enter target container database TDE keystore password:
zdmhost: 2024-02-22T09:27:17.864Z : Processing response file ...
Operation "zdmcli migrate database" scheduled with the job ID "44".
[zdmuser@zdmhost migration]$

[zdmuser@zdmhost migration]$ /u01/app/oracle/product/zdm/bin/zdmcli query job -jobid 44
zdmhost.domain.com: Audit ID: 551
Job ID: 44
User: zdmuser
Client: zdmhost
Job Type: "MIGRATE"
Scheduled job command: "zdmcli migrate database -sourcesid ONPR -rsp /home/zdmuser/migration/zdm_ONPR_physical_online.rsp -sourcenode vmonpr -srcauth zdmauth -srcarg1 user:oracle -srcarg2 identity_file:/home/zdmuser/.ssh/id_rsa -srcarg3 sudo_location:/usr/bin/sudo -targetnode ExaCC-cl01n1 -tgtauth zdmauth -tgtarg1 user:opc -tgtarg2 identity_file:/home/zdmuser/.ssh/id_rsa -tgtarg3 sudo_location:/usr/bin/sudo -tdekeystorepasswd -tgttdekeystorepasswd -pauseafter ZDM_CONFIGURE_DG_SRC"
Scheduled job execution start time: 2024-02-22T10:27:17+01. Equivalent local time: 2024-02-22 10:27:17
Current status: PAUSED
Current Phase: "ZDM_CONFIGURE_DG_SRC"
Result file path: "/u01/app/oracle/chkbase/scheduled/job-44-2024-02-22-10:27:27.log"
Metrics file path: "/u01/app/oracle/chkbase/scheduled/job-44-2024-02-22-10:27:27.json"
Job execution start time: 2024-02-22 10:27:27
Job execution end time: 2024-02-22 10:39:38
Job execution elapsed time: 12 minutes 11 seconds
ZDM_GET_SRC_INFO ................ COMPLETED
ZDM_GET_TGT_INFO ................ COMPLETED
ZDM_PRECHECKS_SRC ............... COMPLETED
ZDM_PRECHECKS_TGT ............... COMPLETED
ZDM_SETUP_SRC ................... COMPLETED
ZDM_SETUP_TGT ................... COMPLETED
ZDM_PREUSERACTIONS .............. COMPLETED
ZDM_PREUSERACTIONS_TGT .......... COMPLETED
ZDM_VALIDATE_SRC ................ COMPLETED
ZDM_VALIDATE_TGT ................ COMPLETED
ZDM_DISCOVER_SRC ................ COMPLETED
ZDM_COPYFILES ................... COMPLETED
ZDM_PREPARE_TGT ................. COMPLETED
ZDM_SETUP_TDE_TGT ............... COMPLETED
ZDM_RESTORE_TGT ................. COMPLETED
ZDM_RECOVER_TGT ................. COMPLETED
ZDM_FINALIZE_TGT ................ COMPLETED
ZDM_CONFIGURE_DG_SRC ............ COMPLETED
ZDM_SWITCHOVER_SRC .............. PENDING
ZDM_SWITCHOVER_TGT .............. PENDING
ZDM_POST_DATABASE_OPEN_TGT ...... PENDING
ZDM_DATAPATCH_TGT ............... PENDING
ZDM_MODIFY_DBUNIQUENAME_TGT ..... PENDING
ZDM_NONCDBTOPDB_PRECHECK ........ PENDING
ZDM_NONCDBTOPDB_CONVERSION ...... PENDING
ZDM_POST_MIGRATE_TGT ............ PENDING
TIMEZONE_UPGRADE_PREPARE_TGT .... PENDING
TIMEZONE_UPGRADE_TGT ............ PENDING
ZDM_POSTUSERACTIONS ............. PENDING
ZDM_POSTUSERACTIONS_TGT ......... PENDING
ZDM_CLEANUP_SRC ................. PENDING
ZDM_CLEANUP_TGT ................. PENDING

Pause After Phase: "ZDM_CONFIGURE_DG_SRC"

We can see that all steps have been completed successfully, and ZDM has paused the migration after the Data Guard has been configured.

I can review full ZDM log:

[zdmuser@zdmhost ~]$ cat /u01/app/oracle/chkbase/scheduled/job-44-2024-02-22-10:27:27.log
zdmhost: 2024-02-22T09:27:27.341Z : Processing response file ...
zdmhost: 2024-02-22T09:27:27.345Z : Processing response file ...
zdmhost: 2024-02-22T09:27:32.418Z : Starting zero downtime migrate operation ...
zdmhost: 2024-02-22T09:27:34.498Z : Executing phase ZDM_GET_SRC_INFO
zdmhost: 2024-02-22T09:27:34.498Z : Retrieving information from source node "vmonpr" ...
zdmhost: 2024-02-22T09:27:34.498Z : retrieving information about database "ONPR" ...
zdmhost: 2024-02-22T09:27:38.793Z : Execution of phase ZDM_GET_SRC_INFO completed
zdmhost: 2024-02-22T09:27:38.819Z : Executing phase ZDM_GET_TGT_INFO
zdmhost: 2024-02-22T09:27:38.819Z : Retrieving information from target node "ExaCC-cl01n1" ...
zdmhost: 2024-02-22T09:27:46.173Z : Determined value for parameter TGT_DATADG is '+DATAC1'
zdmhost: 2024-02-22T09:27:46.173Z : Determined value for parameter TGT_REDODG is '+DATAC1'
zdmhost: 2024-02-22T09:27:46.173Z : Determined value for parameter TGT_RECODG is '+RECOC1'
zdmhost: 2024-02-22T09:27:46.284Z : Execution of phase ZDM_GET_TGT_INFO completed
zdmhost: 2024-02-22T09:27:46.821Z : Executing phase ZDM_PRECHECKS_SRC
zdmhost: 2024-02-22T09:27:46.821Z : Execution of phase ZDM_PRECHECKS_SRC completed
zdmhost: 2024-02-22T09:27:47.080Z : Executing phase ZDM_PRECHECKS_TGT
zdmhost: 2024-02-22T09:27:47.080Z : Execution of phase ZDM_PRECHECKS_TGT completed
zdmhost: 2024-02-22T09:27:47.118Z : Executing phase ZDM_SETUP_SRC
zdmhost: 2024-02-22T09:27:47.118Z : Setting up ZDM on the source node vmonpr ...
vmonpr: 2024-02-22T09:28:49.592Z : TNS aliases successfully setup on the source node vmonpr...
zdmhost: 2024-02-22T09:28:49.694Z : Execution of phase ZDM_SETUP_SRC completed
####################################################################
zdmhost: 2024-02-22T09:28:49.730Z : Executing phase ZDM_SETUP_TGT
zdmhost: 2024-02-22T09:28:49.730Z : Setting up ZDM on the target node ExaCC-cl01n1 ...
ExaCC-cl01n1: 2024-02-22T09:29:12.976Z : TNS aliases successfully setup on the target node ExaCC-cl01n1...
zdmhost: 2024-02-22T09:29:12.979Z : Execution of phase ZDM_SETUP_TGT completed
####################################################################
zdmhost: 2024-02-22T09:29:13.023Z : Executing phase ZDM_VALIDATE_SRC
zdmhost: 2024-02-22T09:29:13.024Z : Validating source environment on node vmonpr ...
vmonpr: 2024-02-22T09:29:23.649Z : Validating SYS account password specified..
vmonpr: 2024-02-22T09:29:34.470Z : Validating source environment...
vmonpr: 2024-02-22T09:29:34.470Z : Ensuring source database is running in ARCHIVELOG mode...
vmonpr: 2024-02-22T09:29:34.871Z : Validating Oracle TDE setup
vmonpr: 2024-02-22T09:29:37.474Z : Validating Oracle Password file
vmonpr: 2024-02-22T09:29:38.476Z : Validating database ONPR role is PRIMARY...
vmonpr: 2024-02-22T09:29:38.478Z : Source environment validated successfully
zdmhost: 2024-02-22T09:29:38.487Z : Execution of phase ZDM_VALIDATE_SRC completed
####################################################################
zdmhost: 2024-02-22T09:29:38.521Z : Executing phase ZDM_VALIDATE_TGT
zdmhost: 2024-02-22T09:29:38.521Z : Validating target environment on node ExaCC-cl01n1 ...
zdmhost: 2024-02-22T09:29:38.573Z : Source database timezone file version 32 is less than target database timezone file version 42. Timezone upgrade operation will be performed on target database after completion of database migration.
ExaCC-cl01n1: 2024-02-22T09:29:50.315Z : Validating specified Oracle ASM storage locations...
ExaCC-cl01n1: 2024-02-22T09:29:54.219Z : validating target database size allocation...
ExaCC-cl01n1: 2024-02-22T09:29:56.922Z : Verifying SQL*Net connectivity to source database ...
ExaCC-cl01n1: 2024-02-22T09:29:57.223Z : verifying passwordless connectivity between target nodes
ExaCC-cl01n1: 2024-02-22T09:29:58.425Z : Target environment validated successfully
zdmhost: 2024-02-22T09:29:58.433Z : Execution of phase ZDM_VALIDATE_TGT completed
####################################################################
zdmhost: 2024-02-22T09:29:58.455Z : Executing phase ZDM_DISCOVER_SRC
zdmhost: 2024-02-22T09:29:58.455Z : Setting up the source node vmonpr for creating standby on the target node ExaCC-cl01n1 ...
vmonpr: 2024-02-22T09:30:09.186Z : Enabling force logging on database ONPR...
vmonpr: 2024-02-22T09:30:09.287Z : Creating standby logs on database ONPR...
vmonpr: 2024-02-22T09:30:13.591Z : Source environment set up successfully
zdmhost: 2024-02-22T09:30:13.700Z : Execution of phase ZDM_DISCOVER_SRC completed
####################################################################
zdmhost: 2024-02-22T09:30:13.729Z : Executing phase ZDM_COPYFILES
zdmhost: 2024-02-22T09:30:13.729Z : Copying files from source node vmonpr to target node ExaCC-cl01n1 ...
vmonpr: 2024-02-22T09:30:24.849Z : Source database "ONPR" credentials exported successfully on node "vmonpr"
zdmhost: 2024-02-22T09:30:29.112Z : Execution of phase ZDM_COPYFILES completed
####################################################################
zdmhost: 2024-02-22T09:30:29.148Z : Executing phase ZDM_PREPARE_TGT
zdmhost: 2024-02-22T09:30:29.148Z : Setting up standby on the target node ExaCC-cl01n1 ...
ExaCC-cl01n1: 2024-02-22T09:31:03.106Z : Target environment set up successfully
zdmhost: 2024-02-22T09:31:03.115Z : Execution of phase ZDM_PREPARE_TGT completed
####################################################################
zdmhost: 2024-02-22T09:31:03.137Z : Executing phase ZDM_SETUP_TDE_TGT
zdmhost: 2024-02-22T09:31:03.137Z : Setting up Oracle Transparent Data Encryption (TDE) keystore on the target node ExaCC-cl01n1 ...
ExaCC-cl01n1: 2024-02-22T09:31:13.880Z : target environment Oracle Transparent Data Encryption (TDE) set up successfully
zdmhost: 2024-02-22T09:31:13.889Z : Execution of phase ZDM_SETUP_TDE_TGT completed
####################################################################
zdmhost: 2024-02-22T09:31:13.913Z : Executing phase ZDM_RESTORE_TGT
zdmhost: 2024-02-22T09:31:13.913Z : Restoring database on the target node ExaCC-cl01n1 ...
ExaCC-cl01n1: 2024-02-22T09:31:36.483Z : database ONPRZ_APP_001T dropped successfully
ExaCC-cl01n1: 2024-02-22T09:31:54.048Z : Target database "ONPRZ_APP_001T" credentials staged successfully on node "ExaCC-cl01n1"
ExaCC-cl01n1: 2024-02-22T09:32:06.392Z : Restoring SPFILE ...
ExaCC-cl01n1: 2024-02-22T09:32:45.923Z : SPFILE restored to /u02/app/oracle/product/19.0.0.0/dbhome_2/dbs/spfileONPRZ_APP_001T1.ora successfully
ExaCC-cl01n1: 2024-02-22T09:32:59.196Z : Restoring control files ...
ExaCC-cl01n1: 2024-02-22T09:33:20.313Z : Control files restored successfully
ExaCC-cl01n1: 2024-02-22T09:33:31.775Z : Restoring and encrypting data files ...
ExaCC-cl01n1: 2024-02-22T09:34:38.628Z : Data files restored and encrypted successfully
ExaCC-cl01n1: 2024-02-22T09:34:38.629Z : Cleaning up any orphaned data ...
ExaCC-cl01n1: 2024-02-22T09:34:38.730Z : Orphaned files clean up successful
ExaCC-cl01n1: 2024-02-22T09:34:39.034Z : Data files restored successfully
ExaCC-cl01n1: 2024-02-22T09:34:51.398Z : Renaming TEMP files and online redo log files ...
ExaCC-cl01n1: 2024-02-22T09:35:03.909Z : TEMP files and online redo log files renamed successfully
ExaCC-cl01n1: 2024-02-22T09:35:16.374Z : Recovering data files ...
ExaCC-cl01n1: 2024-02-22T09:35:20.080Z : Data files recovered successfully
zdmhost: 2024-02-22T09:35:20.094Z : Execution of phase ZDM_RESTORE_TGT completed
####################################################################
zdmhost: 2024-02-22T09:35:20.115Z : Executing phase ZDM_RECOVER_TGT
zdmhost: 2024-02-22T09:35:20.115Z : Recovering database on the target node ExaCC-cl01n1 ...
ExaCC-cl01n1: 2024-02-22T09:35:37.674Z : Target database "ONPRZ_APP_001T" credentials staged successfully on node "ExaCC-cl01n1"
ExaCC-cl01n1: 2024-02-22T09:35:50.018Z : Restoring control files ...
ExaCC-cl01n1: 2024-02-22T09:36:31.849Z : Running RMAN crosscheck on database "ONPRZ_APP_001T" ...
ExaCC-cl01n1: 2024-02-22T09:36:36.354Z : RMAN crosscheck on database "ONPRZ_APP_001T" ran successfully
ExaCC-cl01n1: 2024-02-22T09:36:36.354Z : Running RMAN catalog ...
ExaCC-cl01n1: 2024-02-22T09:36:38.357Z : RMAN catalog ran successfully
ExaCC-cl01n1: 2024-02-22T09:36:38.357Z : Control files restored successfully
ExaCC-cl01n1: 2024-02-22T09:36:50.818Z : Starting incremental restore of data files ...
ExaCC-cl01n1: 2024-02-22T09:36:55.122Z : Incremental restore of data files executed successfully
ExaCC-cl01n1: 2024-02-22T09:36:55.123Z : Cleaning up any orphaned data ...
ExaCC-cl01n1: 2024-02-22T09:36:55.224Z : Orphaned files clean up successful
ExaCC-cl01n1: 2024-02-22T09:36:55.327Z : Data files restored successfully
ExaCC-cl01n1: 2024-02-22T09:37:07.696Z : Renaming TEMP files and online redo log files ...
ExaCC-cl01n1: 2024-02-22T09:37:20.207Z : TEMP files and online redo log files renamed successfully
ExaCC-cl01n1: 2024-02-22T09:37:32.669Z : Recovering data files ...
ExaCC-cl01n1: 2024-02-22T09:37:36.575Z : Data files recovered successfully
zdmhost: 2024-02-22T09:37:36.590Z : Execution of phase ZDM_RECOVER_TGT completed
####################################################################
zdmhost: 2024-02-22T09:37:36.614Z : Executing phase ZDM_FINALIZE_TGT
zdmhost: 2024-02-22T09:37:36.614Z : Finalizing creation of standby database on the target node ExaCC-cl01n1 ...
ExaCC-cl01n1: 2024-02-22T09:37:47.554Z : Updating database cluster resource dependency ...
ExaCC-cl01n1: 2024-02-22T09:38:30.389Z : Creating standby redo logs on target database ONPRZ_APP_001T
ExaCC-cl01n1: 2024-02-22T09:38:30.490Z : Enabling Oracle Data Guard Broker on "ONPRZ_APP_001T" ...
ExaCC-cl01n1: 2024-02-22T09:38:33.493Z : Oracle Data Guard Broker enabled successfully on "ONPRZ_APP_001T"
ExaCC-cl01n1: 2024-02-22T09:38:33.694Z : Target database updated successfully
zdmhost: 2024-02-22T09:38:33.704Z : Execution of phase ZDM_FINALIZE_TGT completed
####################################################################
zdmhost: 2024-02-22T09:38:33.725Z : Executing phase ZDM_CONFIGURE_DG_SRC
zdmhost: 2024-02-22T09:38:33.726Z : Finalize steps done on the source node vmonpr for creating standby on the target node ExaCC-cl01n1 ...
vmonpr: 2024-02-22T09:38:44.648Z : Configuring Oracle Data Guard Broker on "ONPR" ...
vmonpr: 2024-02-22T09:39:38.289Z : Oracle Data Guard Broker configured successfully on "ONPR"
vmonpr: 2024-02-22T09:39:38.390Z : Source database updated successfully
zdmhost: 2024-02-22T09:39:38.398Z : Execution of phase ZDM_CONFIGURE_DG_SRC completed
####################################################################
zdmhost: 2024-02-22T09:39:38.403Z : Job execution paused after phase "ZDM_CONFIGURE_DG_SRC".

We can see ZDM will valide source and target, check SYS password, check and setup TDE, validate Oracle Net connections, validate ASM storage, create standby redo logs, creating standby database using direct transfer data method (the application tablespace will be encrypted), and configure Data Guard.

We can check additional and detailed logs on the source database:

oracle@vmonpr:/home/oracle/ [ONPR] cd  /u00/app/oracle/zdm/zdm_ONPR_44/zdm/log/

oracle@vmonpr:/u00/app/oracle/zdm/zdm_ONPR_44/zdm/log/ [ONPR] ls -ltrh
total 228K
-rw-rw-rw-. 1 oracle dba  14K Feb 22 10:28 zdm_setup_tns_alias_src_12728.log
-rwxrwxrwx. 1 oracle dba    0 Feb 22 10:29 default.log
-rw-rw-rw-. 1 oracle dba  17K Feb 22 10:29 zdm_validate_sys_pass_src_13181.log
-rw-rw-rw-. 1 root   root 57K Feb 22 10:29 zdm_validate_src_13240.log
-rw-rw-rw-. 1 root   root 62K Feb 22 10:30 zdm_discover_src_13722.log
-rw-rw-rw-. 1 root   root 22K Feb 22 10:30 zdm_export_db_cred_src_15024.log
-rw-rw-rw-. 1 oracle dba  43K Feb 22 10:39 zdm_configure_dg_src_18159.log

And on the target (ExaCC):

[opc@ExaCC-cl01n1 ~]$ cd /u02/app/oracle/zdm/zdm_ONPR_RZ2_44/zdm/log/

[opc@ExaCC-cl01n1 log]$ ls -ltrh
total 516K
-rw-rw-rw- 1 oracle root  33K Feb 22 10:29 zdm_setup_tns_alias_tgt_224683.log
-rwxrwxrwx 1 oracle root    0 Feb 22 10:29 default.log
-rw-rw-rw- 1 oracle root  29K Feb 22 10:29 zdm_validate_tgt_231351.log
-rw-rw-rw- 1 root   root  59K Feb 22 10:31 zdm_prepare_tgt_239052.log
-rw-rw-rw- 1 root   root 7.9K Feb 22 10:31 zdm_setup_tde_tgt_242739.log
-rw-rw-rw- 1 root   root  16K Feb 22 10:31 zdm_oss_restore_tgt_dropdatabase_244302.log
-rw-rw-rw- 1 root   root  21K Feb 22 10:31 zdm_import_db_cred_tgt_250414.log
-rw-rw-rw- 1 root   root  24K Feb 22 10:32 zdm_oss_restore_tgt_restoreinit_252706.log
-rw-rw-rw- 1 root   root  19K Feb 22 10:33 zdm_oss_restore_tgt_restorecntrl_260310.log
-rw-rw-rw- 1 root   root  37K Feb 22 10:34 zdm_oss_restore_tgt_restoredb_263322.log
-rw-rw-rw- 1 root   root  36K Feb 22 10:35 zdm_oss_restore_tgt_renametemp_279405.log
-rw-rw-rw- 1 root   root  22K Feb 22 10:35 zdm_oss_restore_tgt_recoverdb_281523.log
-rw-rw-rw- 1 root   root  22K Feb 22 10:35 zdm_import_db_cred_tgt_283595.log
-rw-rw-rw- 1 root   root  28K Feb 22 10:36 zdm_oss_recover_tgt_restorecntrl_288735.log
-rw-rw-rw- 1 root   root  37K Feb 22 10:36 zdm_oss_recover_tgt_restoredb_297044.log
-rw-rw-rw- 1 root   root  36K Feb 22 10:37 zdm_oss_recover_tgt_renametemp_298270.log
-rw-rw-rw- 1 root   root  22K Feb 22 10:37 zdm_oss_recover_tgt_recoverdb_300955.log
-rw-rw-rw- 1 root   root  41K Feb 22 10:38 zdm_finalize_tgt_305026.log

Finally I can check Data Guard configuration and see that my standby is synchronized (no gap).

oracle@vmonpr:/home/oracle/ [ONPR] dgmgrl
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Thu Feb 22 10:42:39 2024
Version 19.10.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.

DGMGRL> connect /
Connected to "ONPR"
Connected as SYSDG.

DGMGRL> show configuration lag

Configuration - ZDM_onpr

  Protection Mode: MaxPerformance
  Members:
  onpr           - Primary database
    onprz_app_001t - Physical standby database
                     Transport Lag:      0 seconds (computed 1 second ago)
                     Apply Lag:          0 seconds (computed 1 second ago)

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 36 seconds ago)

I can see that ZDM created a new instance on the ExaCC, called as the final PDB name:

oracle@ExaCC-cl01n1:/u02/app/oracle/zdm/zdm_ONPR_RZ2_38/zdm/log/ [ONPR1 (CDB$ROOT)] ps -ef | grep [p]mon | grep -i ONPRZ_APP_001T1
oracle   236556      1  0 12:22 ?        00:00:00 ora_pmon_ONPRZ_APP_001T1

If we connect to it, we can see that the instance name is as the final PDB name and the db name is matching the source database one to migrate. The db_unique_name will be the pdb name.

oracle@ExaCC-cl01n1:/u02/app/oracle/product/19.0.0.0/dbhome_2/ [ONPR1 (CDB$ROOT)] export ORACLE_SID=ONPRZ_APP_001T1

oracle@ExaCC-cl01n1:/u02/app/oracle/product/19.0.0.0/dbhome_2/ [ONPRZ_APP_001T1 (CDB$ROOT)] sqh

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Feb 14 13:22:42 2024
Version 19.21.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.21.0.0.0

SQL> set lines 300 pages 500
SQL> select instance_name from v$instance;

INSTANCE_NAME
------------------------------------------------
ONPRZ_APP_001T1

SQL> select name, db_unique_name, open_mode, database_role from v$database;

NAME                        DB_UNIQUE_NAME                                                                             OPEN_MODE                                                    DATABASE_ROLE
--------------------------- ------------------------------------------------------------------------------------------ ------------------------------------------------------------ ------------------------------------------------
ONPR                        ONPRZ_APP_001T                                                                             MOUNTED                                                      PHYSICAL STANDBY

And I could see that this temporary database was single instance database:

SQL> show parameter cluster_database;

NAME                                 TYPE                              VALUE
------------------------------------ --------------------------------- ------------------------------
cluster_database                     boolean                           FALSE
cluster_database_instances           integer                           1

Migration – Maintenance Windows

Now we will have the maintenance windows and switch the database to the ExaCC. After the switchover, ZDM will run datapatch (to patch the database to the new version 19.21), convert the noncdb to pdb, upgrade the timezone and any other post migration tasks.

For this we just need to resume the job. We could even resume it adding a new pause if we want to do each steps separately.

To resume the job:

[zdmuser@zdmhost migration]$ /u01/app/oracle/product/zdm/bin/zdmcli resume job -jobid 44
zdmhost.domain.com: Audit ID: 552

Wrap up

ZDM is really a nice tool, and ZDM will mainly configure all for you. Albeit I was facing sometimes some problem that I had to troubleshoot, I could always find a solution. Oracle ZDM team is also very flexible and available to discuss. Knowing we have all the on-premises databases running 19.10, and as customer does not want to go without fallback, I could unfortunately not test the switchover and next conversion steps. I will blog later on it, once customer databases will be patched to last 19c version, and that I will be able to move forward with the test.