Last week I did some upgrade tests from 19c to 21c. The plan was to copy a 19c PDB to a CDB 21c using Online Clone and then upgrade it to 21c. I have done this many times for 19c as well and just wanted to verify that it still worked with 21c.
Clone PDB to 21c CDB
Since it is a LAB, I gave the user SYSTEM in the CDB19c the appropriate duplicate permissions and then I created from the CDB 21c the appropriate DB link for the clone
SQL*Plus: Release 19.0.0.0.0 - Production on Fri Sep 17 13:17:06 2021 Version 19.3.0.0.0 Copyright (c) 1982, 2019, Oracle. All rights reserved. Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.3.0.0.0 SQL> grant sysoper, create pluggable database to system container=all; Grant succeeded. -- create db link for clone oracle@ora21:/home/oracle/ [cdb01] sqh SQL*Plus: Release 21.0.0.0.0 - Production on Fri Sep 17 13:18:31 2021 Version 21.3.0.0.0 Copyright (c) 1982, 2021, Oracle. All rights reserved. Connected to: Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production Version 21.3.0.0.0 SQL> create database link cdb19c2 connect to system identified by manager using 'cdb19c2'; Database link created.
Now I clone the 19c pdb into the 21c CDB over the dblink
SQL> create pluggable database pdb19c_copy from pdb19c@cdb19c2; Pluggable database created.
Upgrade PDB to 21c
So the next step is to open the PDB and afterwards execute the upgrade script for upgrading the catalog of the pdb to 21c.
SQL> alter pluggable database pdb19c_copy open;
And now with Oracle 21c the magic part happens, or i would say, that a autonoumous Feature found the way to the on prem word.
After a few seconds the opening of the PDB was still not finished, but no error was displayed on the prompt, I looked into the alert log of the database to see what exactly the problem was. When I looked through the entries, I was quite surprised to see that my PDB was in the middle of an upgrade to 21c without Oracle giving me a choice.
A look on the pdb state confirmed what oracle tells me in the alert.log
SQL> show pdbs
CON_ID CON_NAME OPEN MODE RESTRICTED ---------- ------------------------------ ---------- ---------- 2 PDB$SEED READ ONLY NO 3 PDB19c_COPY MIGRATE YES
Here you can see a very interesting snipped of the alert.log
PDB19C_COPY(6):Starting Upgrade on PDB Open 2021-09-17T13:21:09.305411+02:00 PDB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG begin install '19.0.0.0.0' on error capture PDB19C_COPY(6):ORA-65229 signalled during: alter pluggable database application APP$CDB$CATALOG begin install '19.0.0.0.0' on error capture... PDB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG end install '19.0.0.0.0' PDB19C_COPY(6):Completed: alter pluggable database application APP$CDB$CATALOG end install '19.0.0.0.0' PDB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG begin upgrade '19.0.0.0.0' to '21.0.0.0.0.partial' on error capture PDB19C_COPY(6):Completed: alter pluggable database application APP$CDB$CATALOG begin upgrade '19.0.0.0.0' to '21.0.0.0.0.partial' on error capture
On the first line you can see, that oracle has started to upgrade my pdb immediate after i issued the open command.
The next lines are very interesting. If you look at the statements beginning with “alter pluggable database application … ” you can see, that oracle uses Multitenant application container functionality to upgrade the pdb to the new version. It seams, that they manage different versions of the cdb catalog as applications.
With the next command oracle tries to upgrade the application APP$CDB$CATALOG to 19.0.0.0.0 what’s in fact the data dictionary version. This gives an error. I can’t confirm it, but in context of application containers the begin install command always install the newest application version in the pdb. In this context i would expect, that oracle tries to apply the newest 19c patches to my pdb but this failed for some reason.
DB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG begin install '19.0.0.0.0' on error capture PDB19C_COPY(6):ORA-65229 signalled during: alter pluggable database application APP$CDB$CATALOG begin install '19.0.0.0.0' on error capture...
Then the next stage is upgrade of the catalog from version 19.0.0.0.0 to 21.0.0.0.0.partial. The syntax is exactly the same if you upgrade an application pdb to a new version.
PDB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG begin upgrade '19.0.0.0.0' to '21.0.0.0.0.partial' on error capture PDB19C_COPY(6):Completed: alter pluggable database application APP$CDB$CATALOG begin upgrade '19.0.0.0.0' to '21.0.0.0.0.partial' on error capture
And in the last step oracle performs the upgrade from the version 21.0.0.0.0.partial to 21.0.0.0.0. I have no idea why this additional step to 21.0.0.0.0.partial is necessary.
PDB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG begin upgrade '21.0.0.0.0.partial' to '21.0.0.0.0' on error capture PDB19C_COPY(6):Completed: alter pluggable database application APP$CDB$CATALOG begin upgrade '21.0.0.0.0.partial' to '21.0.0.0.0' on error capture
Finally the Upgrade of the Application APP$CDB$CATALOG is done and oracle open the pdb with the new version 21c
PDB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG end upgrade Pluggable database PDB19C_COPY opened read write
Reverse Engineering
To see if my assumptions are correct, we can query the most important app container views in the CDB and the PDB and see what is in there.
Let’s check the applications that are registered in the cdb$root
SQL> select app_name, app_id, app_version, app_capture_module from dba_applications; APP_NAME APP_ID APP_VERSION APP_CAPTURE_MODULE ------------------------------ ---------- ------------------------------ ---------------------------------------------------------------- APP$CDB$SYSTEM 2 1.0 catcon(pid=26657) APP$CDB$PDBONLY$NCDBTOPDB 21 1.0 catcon(pid=26657) APP$CDB$CATALOG 4294967293 21.0.0.0.0 catcon(pid=24698) APP$CDB$CATALOG 4294967292 21.0.0.0.0 catcon(pid=8832) APP$CDB$CATALOG 4294967291 21.0.0.0.0 catcon(pid=1120)
We can see that we have an application APP$CDB$CATALOG that holds the upgrade scripts to upgrade a pdb to 21c and also we have a application APP$CDB$PDBONLY$NCDBTOPDB which is responsible auto upgrading a non cdb to 21c and additionally including the scripts for the conversion from noncdb to pdb
And if we have a look at the app versions, we can see, that oracle stores the catalog upgrade scripts from 12.2 onwards. This allows to auto upgrade the pdb to 21c for all versions beginning with 12.2
SQL> select app_name, app_version from dba_app_versions; APP_NAME APP_VERSION ---------------------------------------- ------------------------------ APP$CDB$SYSTEM 1.0 APP$CDB$PDBONLY$NCDBTOPDB 1.0.upgmode APP$CDB$PDBONLY$NCDBTOPDB 1.0 APP$CDB$CATALOG 12.2.0.1.0 APP$CDB$CATALOG 21.0.0.0.0.partial APP$CDB$CATALOG 21.0.0.0.0 APP$CDB$CATALOG 18.0.0.0.0 APP$CDB$CATALOG 21.0.0.0.0.partial APP$CDB$CATALOG 21.0.0.0.0 APP$CDB$CATALOG 19.0.0.0.0 APP$CDB$CATALOG 21.0.0.0.0.partial APP$CDB$CATALOG 21.0.0.0.0
Replay Upgrade Feature
To learn a bit more about what exactly Oracle is doing here, I talked to Mike Dietrich. He confirmed my experience and told me that the feature implemented with 21c is called “Replay Upgrade”. The process to upgrade a PDB is stored in the CDB from 21c on and as soon as a PDB with a lower version is opened, this recorded process is automatically replayed to upgrade the PDB. This new feature is even rudimentarily documented here.
A big disadvantage with this automatic upgrade process is that no single upgrade log is created for it. In my lab, which consists of an empty PDB, it worked without problems, but for real databases, this is not recommended, since no analysis can be done in case of an error. Also, the replay upgrade does not update the Timezone file to the latest version. This must be done manually afterwards. This was also confirmed by Mike Dietrich and therefore for an upgrade you should always work with the “autoupgrade tool”, which is officially recommended by Oracle.
Based on the documentation, the replay feature in the CDB can also be disabled. I would recommend this, because you have back the full control over upgrades afterwards.
ALTER DATABASE UPGRADE SYNC OFF
To be complete, the feature can be reactivated with the following command.
ALTER DATABASE UPGRADE SYNC ON
Container Compatibility
The Replay Upgrade feature only applies if exactly the same components are installed in the Source CDB and the Target CDB. If this is not the case, the PDB clone cannot be created at all and terminates with an error.
SQL> create pluggable database pdb19_copy from pdb19@cdb19c; create pluggable database pdb19_copy from pdb19@cdb19c * ERROR at line 1: ORA-65346: The PDB version is lower and components (APS, CONTEXT, DV, OLS, ORDIM, SDO, XOQ) are missing in CDB.