Why relocating the PDB after ZDM Physical Online migration with noncdb to cdb conversion is a must?

I have been blogging about ZDM Physical Online migration with noncdb to cdb conversion. As you could read in my previous article, the target database name on CDB level should be the same as the source database, knowing physical online migration is using Data Guard. In our case, at customer, we wanted to group the new migrated PDBs is some CDB, and thus the CDB name was not matching any convention any more. This is why we relocated the new migrated PDB from the target CDB database used during ZDM migration to a final one. I could later realise that in any case, if we are migrating and converting from noncdb to pdb with ZDM it is better to relocate the PDB at the end. Let me explain you why.

ZDM bug on Grid Infra resource

I made some test and migrated a source On-Premise database to the new ExaCC but just stopping after ZDM migration completed. The naming convention will be the same as the previous blog. As we can see the noncdb Source database has been successfully migrated to the new target database on the exaCC.

oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] sqh

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Jul 29 16:45:17 2024
Version 19.22.0.0.0

Copyright (c) 1982, 2023, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.22.0.0.0

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 ONPRZ_APP_001T                 READ WRITE NO

SQL> alter session set container=ONPRZ_APP_001T;

Session altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         3 ONPRZ_APP_001T                 READ WRITE NO

SQL> select status, message from pdb_plug_in_violations;

STATUS     MESSAGE
---------- ------------------------------------------------------------------------------------------------------------------------
RESOLVED   PDB needs to import keys from source.
RESOLVED   Database option RAC mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.
RESOLVED   PDB plugged in is a non-CDB, requires noncdb_to_pdb.sql be run.
PENDING    Database option APS mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.
PENDING    Database option CATJAVA mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.
PENDING    Database option CONTEXT mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.
PENDING    Database option DV mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.
PENDING    Database option JAVAVM mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.
PENDING    Database option OLS mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.
PENDING    Database option ORDIM mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.
PENDING    Database option SDO mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.
PENDING    Database option XML mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.
PENDING    Database option XOQ mismatch: PDB installed version NULL. CDB installed version 19.0.0.0.0.
PENDING    Tablespace SYSTEM is not encrypted. Oracle Cloud mandates all tablespaces should be encrypted.
PENDING    Tablespace SYSAUX is not encrypted. Oracle Cloud mandates all tablespaces should be encrypted.

15 rows selected.

As it was a fast test and as my target database was still on one instance I wanted to configured it to RAC again.

I setup cluster_database instance parameter back to TRUE value.

oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] sqh

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Jul 29 17:20:27 2024
Version 19.22.0.0.0

Copyright (c) 1982, 2023, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.22.0.0.0

SQL> alter system set cluster_database=TRUE scope=spfile sid='*';

System altered.

I stopped my database.

oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] srvctl stop database -d ONPR_CHZ2

Added back the second instance.

oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] srvctl add instance -d ONPR_CHZ2 -i ONPR2 -node ExaCC-cl01n2

Starting the database I could face some errors.

oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] srvctl start database -d ONPR_CHZ2
PRCR-1079 : Failed to start resource ora.onpr_chz2.db
CRS-5017: The resource action "ora.onpr_chz2.db start" encountered the following error:
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u02/app/oracle/product/19.0.0.0/dbhome_1/dbs/initONPR2.ora'
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/ExaCC-cl01n2/crs/trace/crsd_oraagent_oracle.trc".

CRS-2674: Start of 'ora.onpr_chz2.db' on 'ExaCC-cl01n2' failed
CRS-2632: There are no more servers to try to place resource 'ora.onpr_chz2.db' on that would satisfy its placement policy
CRS-5017: The resource action "ora.onpr_chz2.db start" encountered the following error:
ORA-01122: database file 1 failed verification check
ORA-01110: data file 1: '+DATAC5/ONPRZ_APP_001T/datafile/system.651.1175616809'
ORA-01204: file number is 14 rather than 1 - wrong file
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/ExaCC-cl01n1/crs/trace/crsd_oraagent_oracle.trc".

CRS-2674: Start of 'ora.onpr_chz2.db' on 'ExaCC-cl01n1' failed

Why init or spfile not in the ASM? Why the CDB will look in the ZDM temporary noncdb database folder?

I have got my 2 instances, but none are running.

oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] srvctl status database -d ONPR_CHZ2
Instance ONPR1 is not running on node ExaCC-cl01n1
Instance ONPR2 is not running on node ExaCC-cl01n2

I checked the grid resource configuration and could realise that ZDM changed the spfile to the ZDM temporary noncdb database one.

oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] srvctl config database -d ONPR_CHZ2
Database unique name: ONPR_CHZ2
Database name: ONPR
Oracle home: /u02/app/oracle/product/19.0.0.0/dbhome_1
Oracle user: oracle
Spfile: /u02/app/oracle/product/19.0.0.0/dbhome_1/dbs/spfileONPRZ_APP_001T1.ora
Password file: +DATAC5/ONPR_CHZ2/PASSWORD/pwdonpr_chz2.599.1175606525
Domain: domain.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATAC5
Mount point paths: /acfs01
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: racoper
Database instances: ONPR1,ONPR2
Configured nodes: ExaCC-cl01n1,ExaCC-cl01n2
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed

I checked the spfile for the target CDB in the ASM.

[grid@ExaCC-cl01n1 ~]$ asmcmd
ASMCMD> ls +DATAC5/ONPR_CHZ2/PARAMETERFILE/
spfile.621.1175607313

And updated the grid infra resource to match it.

oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] srvctl modify database -d ONPR_CHZ2 -spfile '+DATAC5/ONPR_CHZ2/PARAMETERFILE/spfile.621.1175607313'
oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] srvctl start database -d ONPR_CHZ2
oracle@ExaCC-cl01n1:~/ [ONPR1 (CDB$ROOT)] srvctl status database -d ONPR_CHZ2
Instance ONPR1 is running on node ExaCC-cl01n1
Instance ONPR2 is running on node ExaCC-cl01n2

Placement of the PDB datafiles

As we can see from the ZDM log, ZDM is doing the noncdb to pdb conversion by creating a pluggable database with the option NOCOPY.

oracle@ExaCC-cl01n1:/u02/app/oracle/zdm/zdm_ONPR_CHZ2_280/zdm/log/ [ONPR1] grep -i "CREATE PLUGGABLE DATABASE" zdm_noncdbtopdb_conversion_211779.log
CREATE PLUGGABLE DATABASE ONPRZ_APP_001T AS CLONE USING '/tmp/ONPRZ_APP_001T.xml' NOCOPY TEMPFILE REUSE

This means the PDB datafiles remains at the same position as the noncdb database was having, in our case ONPRZ_APP_001T.

So the new PDB ONPRZ_APP_001T is located in ONPR_CHZ2 CDB but having its datafiles not in one subdirectory from the CDB using the GUID from the PDB but an external folder.

Location of the PDB datafiles:

[grid@ExaCC-cl01n1 ~]$ asmcmd
ASMCMD> ls DATAC5/ONPRZ_APP_001T/
CHANGETRACKING/
DATAGUARDCONFIG/
TEMPFILE/
controlfile/
datafile/
onlinelog/
password/
ASMCMD> ls DATAC5/ONPRZ_APP_001T/datafile
APP1.646.1175616827
APP2.644.1175616827
SYSAUX.650.1175616809
SYSTEM.651.1175616809
UNDOTBS1.648.1175616825
USERS.647.1175616827

Location of the CDB and other PDB datafiles:

ASMCMD> ls DATAC5/ONPR_CHZ2
0F39DE50B35B3B53E0638A5346646D89/
1E622346BAF56871E0630E1FA10AEBC4/
1E64CA31F0169FAEE0630E1FA10A0290/
CONTROLFILE/
DATAFILE/
ONLINELOG/
PARAMETERFILE/
PASSWORD/
TEMPFILE/
ASMCMD> ls DATAC5/ONPR_CHZ2/1E64CA31F0169FAEE0630E1FA10A0290/
TEMPFILE/
ASMCMD> ls DATAC5/ONPR_CHZ2/1E622346BAF56871E0630E1FA10AEBC4/
TEMPFILE/
ASMCMD> ls DATAC5/ONPR_CHZ2/F39DE50B35B3B53E0638A5346646D89/DATAFILE/
SYSAUX.613.1175606819
SYSTEM.612.1175606819
UNDOTBS1.614.1175606819

To wrap up

In case of a ZDM Physical Online Migration using noncdb to pdb conversion I would recommend to relocate the PDB at the end of the migration.