In an oracle environment with Transparent Data Encryption (TDE) the transfer of PDB may require additional steps.

In this blog we will see some steps which may be required when doing Unplug and Plug  of pluggable database

We plan to unplug the pluggable database PDB1 from PROD01 and plug it into CLNE02. Both container have TDE configured

Source : PROD01  with TDE

10:37:38 SQL> show PDB1s

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB1$SEED                       READ ONLY  NO
         3 PDB1                            READ WRITE NO
10:37:43 SQL>

Target : CLNE02 with TDE

10:37:05 SQL> show PDB1s

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB1$SEED                       READ ONLY  NO
	 3 POD2                           READ WRITE NO
10:37:09 SQL>

Export the encryption key on the source PDB1

On the source container PROD01, we have to export the encryption key while connecting to PDB1.

The option force is needed if AUTO LOGIN is used

SQL> alter session set container=PDB1;

Session altered.


11:24:23 SQL> ADMINISTER KEY MANAGEMENT EXPORT ENCRYPTION KEYS WITH SECRET "********" TO '/home/oracle/tde/export.p12' FORCE keystore IDENTIFIED BY ********;

keystore altered.

11:24:25 SQL>

Then verify that the key was exported.

11:24:25 SQL> host ls -l /home/oracle/tde/export.p12
-rw-r--r--. 1 oracle oinstall 2612 Feb 28 11:24 /home/oracle/tde/export.p12

11:25:36 SQL>

Unplug PDB1 from PROD01

After exporting the key, we can now unplug  PDB1

11:28:08 SQL> show con_name

CON_NAME
------------------------------
CDB$ROOT

11:28:17 SQL> alter pluggable database PDB1 close;

Pluggable database altered.

11:28:30 SQL> show PDB1s

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB1$SEED                       READ ONLY  NO
         3 PDB1                           MOUNTED
         5 PDB1CLONE                      MOUNTED
11:28:36 SQL>


11:28:36 SQL> ALTER PLUGGABLE DATABASE PDB1 UNPLUG INTO '/home/oracle/PDB1.xml';

Pluggable database altered.

11:30:54 SQL>

Plug the database on CLNE02 

Now plug  PDB1 into CLNE02 

SQL> show parameter db_name

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_name                              string      CLNE02
SQL> show con_name

CON_NAME
------------------------------
CDB$ROOT

SQL> CREATE PLUGGABLE DATABASE PDB1 USING '/home/oracle/PDB1.xml';

Pluggable database created.

SQL>

At this level we can open PDB1 in CLNE02 with errors, which is normal as we need to import the encryption key

11:37:16 SQL> alter pluggable database PDB1 open;

Warning: PDB1 altered with errors.       
11:37:34 SQL>

Import the key into PDB1 on CLNE02

Now let’s import the exported key into PDB1 on CLNE02.

-Open PDB1 in R/W mode if not

13:13:18 SQL> alter pluggable database PDB1 open;

Warning: PDB1 altered with errors.

13:13:35 SQL> show PDB1s

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB1$SEED                       READ ONLY  NO
         3 POD2                           READ WRITE NO
         4 PDB1                           READ WRITE YES

-Open the keystore on PDB if not

13:13:46 SQL> alter session set container=PDB1;


13:16:55 SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY ********;

keystore altered.

13:16:59 SQL>

-And then import the key

SQL> ADMINISTER KEY MANAGEMENT IMPORT ENCRYPTION KEYS WITH SECRET ******** FROM '/home/oracle/tde/export.p12' IDENTIFIED BY ******** WITH BACKUP;


keystore altered.

-On CLNE02, restart the pluggable database PDB1

SQL> show con_name

CON_NAME
------------------------------
PDB1

13:18:52 SQL> shutdown
Pluggable Database closed.

13:19:00 SQL> startup
Pluggable Database opened.

PDB1 should no longer return any error now when starting in open mode

SQL> conn / as sysdba
Connected.
13:21:56 SQL> alter pluggable database PDB1 close;

Pluggable database altered.

13:22:07 SQL> alter pluggable database PDB1 open;

Pluggable database altered.

13:22:14 SQL> alter pluggable database all save state;

Pluggable database altered.

We can then validate on PDB1 that encrypted objects are accessible.

Conclusion

Hope this blog may help

L’article Unplug/Plug PDB in TDE Environment est apparu en premier sur dbi Blog.

From the CDB$ROOT, create a common user and configure the database to be ready to extract data via GoldenGate:

SQL> create user c##gg_admin identified by "*****" default tablespace goldengate temporary tablespace temp;

User created.

SQL>

SQL> alter user c##gg_admin quota unlimited on goldengate;

User altered.

SQL>


SQL> grant create session, connect,resource,alter system, select any dictionary, flashback any table to c##gg_admin container=all;

Grant succeeded.

SQL>

SQL> exec dbms_goldengate_auth.grant_admin_privilege(grantee => 'c##gg_admin',container=>'all');

PL/SQL procedure successfully completed.

SQL> alter user c##gg_admin set container_data=all container=current;

User altered.

SQL>

SQL> grant alter any table to c##gg_admin container=ALL;

Grant succeeded.

SQL>

alter system set enable_goldengate_replication=true scope=both;


SQL> alter database force logging;


SQL> alter pluggable database add supplemental log data;

Pluggable database altered.

SQL>

Add the schematrandata for the schema concerned:

GGSCI (vmld-01726 as c##gg_admin@MYCDB) 3> add schematrandata schema_source

2022-04-13 18:06:55  INFO    OGG-01788  SCHEMATRANDATA has been added on schema "schema_source".

2022-04-13 18:06:55  INFO    OGG-01976  SCHEMATRANDATA for scheduling columns has been added on schema "schema_source".

2022-04-13 18:06:55  INFO    OGG-10154  Schema level PREPARECSN set to mode NOWAIT on schema "schema_source".

2022-04-13 18:07:00  INFO    OGG-10471  ***** Oracle Goldengate support information on table schema_source.ZZ_DUMMY *****
Oracle Goldengate support native capture on table schema_source.ZZ_DUMMY.
Oracle Goldengate marked following column as key columns on table schema_source.ZZ_DUMMY: SCN, D, COMMENT_TXT
No unique key is defined for table schema_source.ZZ_DUMMY.

2022-04-13 18:07:00  INFO    OGG-10471  ***** Oracle Goldengate support information on table schema_source.ZZ_DUMMY2 *****
Oracle Goldengate support native capture on table schema_source.ZZ_DUMMY2.
Oracle Goldengate marked following column as key columns on table schema_source.ZZ_DUMMY2: SCN, D, COMMENT_TXT
No unique key is defined for table schema_source.ZZ_DUMMY2.

2022-04-13 18:07:00  INFO    OGG-10471  ***** Oracle Goldengate support information on table schema_source.ZZ_SURVEILLANCE *****
Oracle Goldengate support native capture on table schema_source.ZZ_SURVEILLANCE.
Oracle Goldengate marked following column as key columns on table schema_source.ZZ_SURVEILLANCE: I.

2022-04-13 18:07:00  INFO    OGG-10471  ***** Oracle Goldengate support information on table schema_source.ZZ_SURVEILLANCE_COPY *****
Oracle Goldengate support native capture on table schema_source.ZZ_SURVEILLANCE_COPY.
Oracle Goldengate marked following column as key columns on table schema_source.ZZ_SURVEILLANCE_COPY: I, SURV_DATE, ELLAPSED_1, ELLAPSED_2, CLIENT_HOST, CLIENT_TERMINAL, OS_USER, CLIENT_PROGRAM, INFO
No unique key is defined for table schema_source.ZZ_SURVEILLANCE_COPY.

GGSCI (vmld-01726 as c##gg_admin@MYCDB/CDB$ROOT) 3> dblogin userid c##gg_admin@MYPDB password xxxx
Successfully logged into database.

GGSCI (vmld-01726 as c##gg_admin@MYCDB) 4> info schematrandata schema_source

2022-04-13 18:32:43  INFO    OGG-06480  Schema level supplemental logging, excluding non-validated keys, is enabled on schema "schema_source".

2022-04-13 18:32:43  INFO    OGG-01980  Schema level supplemental logging is enabled on schema "schema_source" for all scheduling columns.

2022-04-13 18:32:43  INFO    OGG-10462  Schema "schema_source" have 4 prepared tables for instantiation.

GGSCI (vmld-01726 as c##gg_admin@MYCDB) 5>

Create a new alias connection to the container database and register the extract, the extract must be registered into the root container (CDB$ROOT) even the data to capture are from the PDB:

GGSCI (myserver) 10> alter credentialstore add user c##gg_admin@MYCDB_X1 alias ggadmin_exacc
Password:

Credential store altered.

GGSCI (myserver) 11> dblogin useridalias ggadmin
Successfully logged into database CDB$ROOT.

GGSCI (myserver as c##gg_admin@MYCDB/CDB$ROOT) 2>

GGSCI (myserver as c##gg_admin@MYCDB/CDB$ROOT) 2> register extract E3 database container (MYPDB)

2022-04-13 18:31:19  INFO    OGG-02003  Extract E3 successfully registered with database at SCN 3386436450080


GGSCI (myserver as c##gg_admin@MYCDB/CDB$ROOT) 3>

Save the SCN –> 3386436450080

Create the EXTRACT, connected on the CDB:

[oracle@myserver:/u01/app/oracle/product/19.1.0.0.4/gg_1]$ mkdir -p /u01/gs_x/ogg/

GGSCI (myserver) 7> add extract E3, integrated tranlog, begin now
EXTRACT (Integrated) added.


GGSCI (myserver) 8> INFO ALL

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     STOPPED     E3		    00:00:00      00:00:06


GGSCI (myserver) 9>


GGSCI (myserver) 9> add exttrail /u01/gs_x/ogg/gz, extract E3
EXTTRAIL added.


GGSCI (myserver) 2> edit param E3


GGSCI (myserver as c##gg_admin@MYCDB/CDB$ROOT) 13> edit param E3
Extract E3
useridalias ggadmin
Exttrail /u01/gs_x/ogg/gz
LOGALLSUPCOLS
UPDATERECORDFORMAT COMPACT
DDL  &
INCLUDE MAPPED OBJNAME MYPDB.SCHEMA.*
Sequence MYPDB.SCHEMA.*;
Table MYPDB.SCHEMA.* ;

The parameter Table must be prefixed by the Pdb Name

Start the Extract always from the CDB$ROOT:

GGSCI (myserver as c##gg_admin@MY_CDB/CDB$ROOT) 12> START EXTRACT E3 atcsn 3386436450080

Sending START request to MANAGER ...
EXTRACT E3 starting


GGSCI (myserver as c##gg_admin@MYCDB/CDB$ROOT) 15> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     E3      00:00:05      00:00:03

Check the extract is running.
Now you are ready to create the Pump process, do the Initial Load and create the replicat process on the target.

L’article How to create an Oracle GoldenGate EXTRACT in Multitenant est apparu en premier sur dbi Blog.

Clone PDB to 21c CDB
Since it is a LAB, I gave the user SYSTEM in the CDB19c the appropriate duplicate permissions and then I created from the CDB 21c the appropriate DB link for the clone

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Sep 17 13:17:06 2021
Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
SQL> grant sysoper, create pluggable database to system container=all;
Grant succeeded.
-- create db link for clone
oracle@ora21:/home/oracle/ [cdb01] sqh
SQL*Plus: Release 21.0.0.0.0 - Production on Fri Sep 17 13:18:31 2021
Version 21.3.0.0.0
Copyright (c) 1982, 2021, Oracle.  All rights reserved.
Connected to:
Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0
SQL> create database link cdb19c2 connect to system identified by manager using 'cdb19c2';
Database link created.

Now I clone the 19c pdb into the 21c CDB over the dblink

SQL> create pluggable database pdb19c_copy from pdb19c@cdb19c2;
Pluggable database created.

Upgrade PDB to 21c
So the next step is to open the PDB and afterwards execute the upgrade script for upgrading the catalog of the pdb to 21c.

SQL> alter pluggable database pdb19c_copy open;

And now with Oracle 21c the magic part happens, or i would say, that a autonoumous Feature found the way to the on prem word.
After a few seconds the opening of the PDB was still not finished, but no error was displayed on the prompt, I looked into the alert log of the database to see what exactly the problem was. When I looked through the entries, I was quite surprised to see that my PDB was in the middle of an upgrade to 21c without Oracle giving me a choice.
A look on the pdb state confirmed what oracle tells me in the alert.log

SQL> show pdbs

CON_ID     CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
2          PDB$SEED                       READ ONLY NO
3          PDB19c_COPY                    MIGRATE   YES

Here you can see a very interesting snipped of the alert.log

PDB19C_COPY(6):Starting Upgrade on PDB Open
2021-09-17T13:21:09.305411+02:00
PDB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG begin install '19.0.0.0.0' on error capture
PDB19C_COPY(6):ORA-65229 signalled during: alter pluggable database application APP$CDB$CATALOG begin install '19.0.0.0.0' on error capture...
PDB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG end install '19.0.0.0.0'
PDB19C_COPY(6):Completed: alter pluggable database application APP$CDB$CATALOG end install '19.0.0.0.0'
PDB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG begin upgrade
'19.0.0.0.0' to '21.0.0.0.0.partial' on error capture
PDB19C_COPY(6):Completed: alter pluggable database application APP$CDB$CATALOG begin upgrade
'19.0.0.0.0' to '21.0.0.0.0.partial' on error capture

On the first line you can see, that oracle has started to upgrade my pdb immediate after i issued the open command.
The next lines are very interesting. If you look at the statements beginning with “alter pluggable database application … ” you can see, that oracle uses Multitenant application container functionality to upgrade the pdb to the new version. It seams, that they manage different versions of the cdb catalog as applications.

With the next command oracle tries to upgrade the application APP$CDB$CATALOG to 19.0.0.0.0 what’s in fact the data dictionary version. This gives an error. I can’t confirm it, but in context of application containers the begin install command always install the newest application version in the pdb. In this context i would expect, that oracle tries to apply the newest 19c patches to my pdb but this failed for some reason.

DB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG begin install '19.0.0.0.0' on error capture 
PDB19C_COPY(6):ORA-65229 signalled during: alter pluggable database application APP$CDB$CATALOG begin install '19.0.0.0.0' on error capture...

Then the next stage is upgrade of the catalog from version 19.0.0.0.0 to 21.0.0.0.0.partial. The syntax is exactly the same if you upgrade an application pdb to a new version.

PDB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG begin upgrade 
'19.0.0.0.0' to '21.0.0.0.0.partial' on error capture 
PDB19C_COPY(6):Completed: alter pluggable database application APP$CDB$CATALOG begin upgrade '19.0.0.0.0' to '21.0.0.0.0.partial' on error capture

And in the last step oracle performs the upgrade from the version 21.0.0.0.0.partial to 21.0.0.0.0. I have no idea why this additional step to 21.0.0.0.0.partial is necessary.

PDB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG begin upgrade '21.0.0.0.0.partial' to '21.0.0.0.0' on error capture
PDB19C_COPY(6):Completed: alter pluggable database application APP$CDB$CATALOG begin upgrade '21.0.0.0.0.partial' to '21.0.0.0.0' on error capture

Finally the Upgrade of the Application APP$CDB$CATALOG is done and oracle open the pdb with the new version 21c

PDB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG end upgrade
Pluggable database PDB19C_COPY opened read write

Reverse Engineering
To see if my assumptions are correct, we can query the most important app container views in the CDB and the PDB and see what is in there.
Let’s check the applications that are registered in the cdb$root

SQL> select app_name, app_id, app_version, app_capture_module from dba_applications;

APP_NAME                           APP_ID APP_VERSION                    APP_CAPTURE_MODULE
------------------------------ ---------- ------------------------------ ----------------------------------------------------------------
APP$CDB$SYSTEM                          2 1.0                            catcon(pid=26657)
APP$CDB$PDBONLY$NCDBTOPDB              21 1.0                            catcon(pid=26657)
APP$CDB$CATALOG                4294967293 21.0.0.0.0                     catcon(pid=24698)
APP$CDB$CATALOG                4294967292 21.0.0.0.0                     catcon(pid=8832)
APP$CDB$CATALOG                4294967291 21.0.0.0.0                     catcon(pid=1120)

We can see that we have an application APP$CDB$CATALOG that holds the upgrade scripts to upgrade a pdb to 21c and also we have a application APP$CDB$PDBONLY$NCDBTOPDB which is responsible auto upgrading a non cdb to 21c and additionally including the scripts for the conversion from noncdb to pdb
And if we have a look at the app versions, we can see, that oracle stores the catalog upgrade scripts from 12.2 onwards. This allows to auto upgrade the pdb to 21c for all versions beginning with 12.2

SQL> select app_name, app_version from dba_app_versions;

APP_NAME                                 APP_VERSION
---------------------------------------- ------------------------------
APP$CDB$SYSTEM                           1.0
APP$CDB$PDBONLY$NCDBTOPDB                1.0.upgmode
APP$CDB$PDBONLY$NCDBTOPDB                1.0
APP$CDB$CATALOG                          12.2.0.1.0
APP$CDB$CATALOG                          21.0.0.0.0.partial
APP$CDB$CATALOG                          21.0.0.0.0
APP$CDB$CATALOG                          18.0.0.0.0
APP$CDB$CATALOG                          21.0.0.0.0.partial
APP$CDB$CATALOG                          21.0.0.0.0
APP$CDB$CATALOG                          19.0.0.0.0
APP$CDB$CATALOG                          21.0.0.0.0.partial
APP$CDB$CATALOG                          21.0.0.0.0

Replay Upgrade Feature
To learn a bit more about what exactly Oracle is doing here, I talked to Mike Dietrich. He confirmed my experience and told me that the feature implemented with 21c is called “Replay Upgrade”. The process to upgrade a PDB is stored in the CDB from 21c on and as soon as a PDB with a lower version is opened, this recorded process is automatically replayed to upgrade the PDB. This new feature is even rudimentarily documented here.
A big disadvantage with this automatic upgrade process is that no single upgrade log is created for it. In my lab, which consists of an empty PDB, it worked without problems, but for real databases, this is not recommended, since no analysis can be done in case of an error. Also, the replay upgrade does not update the Timezone file to the latest version. This must be done manually afterwards. This was also confirmed by Mike Dietrich and therefore for an upgrade you should always work with the “autoupgrade tool”, which is officially recommended by Oracle.
Based on the documentation, the replay feature in the CDB can also be disabled. I would recommend this, because you have back the full control over upgrades afterwards.

ALTER DATABASE UPGRADE SYNC OFF

To be complete, the feature can be reactivated with the following command.

ALTER DATABASE UPGRADE SYNC ON

Container Compatibility
The Replay Upgrade feature only applies if exactly the same components are installed in the Source CDB and the Target CDB. If this is not the case, the PDB clone cannot be created at all and terminates with an error.

SQL> create pluggable database pdb19_copy from pdb19@cdb19c;
create pluggable database pdb19_copy from pdb19@cdb19c
*
ERROR at line 1:
ORA-65346: The PDB version is lower and components (APS, CONTEXT, DV, OLS,
ORDIM, SDO, XOQ) are missing in CDB.

L’article Magic Upgrade to 21c with Replay Upgrade Feature est apparu en premier sur dbi Blog.

SQL> show pdbs
   CON_ID     CON_NAME    OPEN MODE    RESTRICTED
_________ ____________ ____________ _____________
        2 PDB$SEED     READ ONLY    NO
        3 CDB1PDB01    MOUNTED
        4 CDB1PDB03    MOUNTED
        5 CDB1PDB02    MOUNTED

I want to create a 4th one:

SQL> create pluggable database CDB1PDB04 from CDB1PDB03;

create pluggable database CDB1PDB04 from CDB1PDB03
                          *
ERROR at line 1:
ORA-65010: maximum number of pluggable databases created

It fails. The maximum number of pluggable databases is defined by MAX_PDBS, but I defined nothing in my SPFILE:

SQL> show spparameter max_pdbs
SID NAME     TYPE    VALUE
--- -------- ------- -----
*   max_pdbs integer

I thought that the default was 4098 (which is incorrect anyway as you cannot create more than 4096) but it is actually 5 here:

SQL> show parameter max_pdbs
NAME     TYPE    VALUE
-------- ------- -----
max_pdbs integer 5

Ok… this parameter is supposed to count the number of user pluggable databases (the ones with CON_ID>2) and I have 3 of them here. The limit is 5 and I have an error mentioning that I’ve reached the limit. That’s not the first time I see wrong maths with this parameter. But there’s worse as I cannot change it:

SQL> alter system set max_pdbs=6;

alter system set max_pdbs=6
 *
ERROR at line 1:
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-65334: invalid number of PDBs specified

I can change it in the SPFILE but it doesn’t help me to create more pluggable databases:

SQL> alter system set max_pdbs=200 scope=spfile;

System altered.

SQL> startup force;

Total System Global Area   2147482744 bytes
Fixed Size                    9137272 bytes
Variable Size               587202560 bytes
Database Buffers           1543503872 bytes
Redo Buffers                  7639040 bytes
Database mounted.
Database opened.

SQL> show parameter max_pdbs
NAME     TYPE    VALUE
-------- ------- -----
max_pdbs integer 200

SQL> create pluggable database CDB1PDB04 from CDB1PDB03;

create pluggable database CDB1PDB04 from CDB1PDB03
                          *
ERROR at line 1:
ORA-65010: maximum number of pluggable databases created

Something bridles me. There’s a MOS Note ORA-65010 When Oracle Database Hosted on AWS Cloud (Doc ID 2328600.1) about the same problem but that’s in 12.1.0.2 (before MAX_PDBS was introduced) which is supposed to be fixed in AUG 2017 PSU. But here I am 3 years later in 19.6 (the January 2020 Release Update for the latest version available on-premises).

So, Oracle limits the number of pluggable databases when we are on a public cloud provider which is not the Oracle Public Cloud. This limitation is not documented in the licensing documentation which mentions 252 as the Enterprise Edition limit, and I see nothing about “Authorized Cloud Environments” limitations for this item. This, and the fact that it can come and go with Release Updates put customers at risk when running on AWS EC2: financial risk and availability risk. I think there are only two choices, on long term, when you want to run your database on a cloud: go to Oracle Cloud or leave for another Database.

How does the Oracle instance know on which public cloud you run? All cloud platforms provide some metadata through HTTP api. I have straced all sendto() and recvfrom() system calls when starting the instance:

strace -k -e trace=recvfrom,sendto -yy -s 1000 -f -o trace.trc sqlplus / as sysdba <<<'startup force'

And I searched for Amazon and AWS here:

This is clear: the instance has a function to detect the cloud provider (kgcs_clouddb_provider_detect) when initializing the SGA in a multitenant architecture (kpdbInitSga) with the purpose of detecting non-oracle clouds (kscs_is_non_oracle_cloud). This queries the AWS metadata (documented on Retrieving Instance Metadata):

[oracle@ora-cdb-1 ~]$ curl http://169.254.169.254/latest/meta-data/services/domain
amazonaws.com/

When Oracle software sees the name of the enemy in the domain name amazonaws.com, it sets an internal limit for the number of pluggable databases that overrides the MAX_PDBS setting. Ok, I don’t need this metadata and I’m root on EC2 so my simple workaround is to block this metadata API:

[root@ora-cdb-1 ~]# iptables -A OUTPUT -d 169.254.169.254  -j REJECT
[root@ora-cdb-1 ~]# iptables -L
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
REJECT     udp  --  anywhere             10.0.0.2             udp dpt:domain reject-with icmp-port-unreachable
REJECT     all  --  anywhere             10.0.0.2             reject-with icmp-port-unreachable

Then restart the instance and it works: I can set or reset MAX_PDBS and create more pluggable databases.

I can remove the rule

[root@ora-cdb-1 ~]# iptables -D OUTPUT -d 169.254.169.254  -j REJECT

If, for watever reason I want to revert back.

Finally, because they had many bugs with the MAX_PDBS soft limit, there’s a parameter to disable it and this disables also the hard limit:

SQL> alter system set "_cdb_disable_pdb_limit"=true scope=spfile;
System altered.

Thanks to Mauricio Melnik for the heads-up on that:

MOS 2538172.1 _cdb_disable_pdb_limit=true …. maybe?

— MM (@maurimelnik) March 26, 2020

However, with this parameter you cannot control anymore the maximum number of PDBs so don’t forget to monitor your AUX_COUNT in DBA_FEATURE_USAGE_STATISTICS.

Here was my discovery when preparing the multitenant workshop lab environment. Note that given the current situation where everybody works from home when possible, we are ready to give this training full of hands-on exercises though Microsoft Teams and AWS EC2 virtual machines. Two days to be comfortable when moving to CDB architecture, which is what should be done this year when you plan to stay with Oracle Database for the future versions.

Update 27-MAR-2020

In order not to sound too negative here, this limit on AWS platforms has been removed in the past and this may be a bug re-introduced with the change from 1 to 3 PDBs in Standard Edition.

L’article Oracle disables your multitenant option when you run on EC2 est apparu en premier sur dbi Blog.

.
Creating – and using – your own services has always been the recommendation. You can connect to a database without a service name, though the instance SID, but this is not what you should do. Each database registers its db_unique_name as a service, and you can use it to connect, but it is always better to create your own application service(s). In multitenant, each PDB registers its name as a service, but the recommendation is still there: create your own services, and connect with your services.
I’ll show in this blog post what happens if you use the PDB name as a service and the standby database registers to the same listener as the primary database. Of course, you can workaround the non-unique service names by registering to different listeners. But this just hides the problem. The main reason to use services is to be independent from physical attributes, so being forced to assign a specific TCP/IP port is not better than using an instance SID.

I have the primary (CDB1) and standby (CDB2) databases registered to the default local listener:

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 03-FEB-2018 23:11:23
 
Copyright (c) 1991, 2016, Oracle.  All rights reserved.
 
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                02-FEB-2018 09:32:30
Uptime                    1 days 13 hr. 38 min. 52 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/VM122/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=VM122)(PORT=5501))(Presentation=HTTP)(Session=RAW))
Services Summary...
Service "627f7512a0452fd4e0537a38a8c055c0" has 2 instance(s).
  Instance "CDB1", status READY, has 1 handler(s) for this service...
  Instance "CDB2", status READY, has 1 handler(s) for this service...
Service "CDB1" has 1 instance(s).
  Instance "CDB1", status READY, has 1 handler(s) for this service...
Service "CDB1XDB" has 1 instance(s).
  Instance "CDB1", status READY, has 1 handler(s) for this service...
Service "CDB1_CFG" has 2 instance(s).
  Instance "CDB1", status READY, has 1 handler(s) for this service...
  Instance "CDB2", status READY, has 1 handler(s) for this service...
Service "CDB1_DGB" has 1 instance(s).
  Instance "CDB1", status READY, has 1 handler(s) for this service...
Service "CDB1_DGMGRL" has 1 instance(s).
  Instance "CDB1", status UNKNOWN, has 1 handler(s) for this service...
Service "CDB2" has 1 instance(s).
  Instance "CDB2", status READY, has 1 handler(s) for this service...
Service "CDB2XDB" has 1 instance(s).
  Instance "CDB2", status READY, has 1 handler(s) for this service...
Service "CDB2_DGB" has 1 instance(s).
  Instance "CDB2", status READY, has 1 handler(s) for this service...
Service "CDB2_DGMGRL" has 1 instance(s).
  Instance "CDB2", status UNKNOWN, has 1 handler(s) for this service...
Service "pdb1" has 2 instance(s).
  Instance "CDB1", status READY, has 1 handler(s) for this service...
  Instance "CDB2", status READY, has 1 handler(s) for this service...
The command completed successfully

Look at service ‘pdb1’, which is the name for my PDB. Connecting to //localhost:1521/PDB1 can connect you randomly to CDB1 (the primary database) or CDB2 (the standby database).

Here is an example, connecting several times to the PDB1 service:

[oracle@VM122 ~]$ for i in {1..5} ; do sqlplus -L -s sys/oracle@//localhost/pdb1 as sysdba <<< 'select name,open_mode,instance_name from v$instance , v$database;'; done
 
NAME      OPEN_MODE            INSTANCE_NAME
--------- -------------------- ----------------
CDB1      READ WRITE           CDB1
 
NAME      OPEN_MODE            INSTANCE_NAME
--------- -------------------- ----------------
CDB1      READ ONLY WITH APPLY CDB2
 
NAME      OPEN_MODE            INSTANCE_NAME
--------- -------------------- ----------------
CDB1      READ WRITE           CDB1
 
NAME      OPEN_MODE            INSTANCE_NAME
--------- -------------------- ----------------
CDB1      READ ONLY WITH APPLY CDB2
 
NAME      OPEN_MODE            INSTANCE_NAME
--------- -------------------- ----------------
CDB1      READ WRITE           CDB1

I was connected at random to CDB1 or CDB2.

As an administrator, you know the instance names and you can connect to the one you want with: //localhost:1521/PDB1/CDB1 or //localhost:1521/PDB1/CDB2:

[oracle@VM122 ~]$ for i in {1..3} ; do sqlplus -L -s sys/oracle@//localhost/pdb1/CDB1 as sysdba <<< 'select name,open_mode,instance_name from v$instance , v$database;'; done
 
NAME      OPEN_MODE            INSTANCE_NAME
--------- -------------------- ----------------
CDB1      READ WRITE           CDB1
 
NAME      OPEN_MODE            INSTANCE_NAME
--------- -------------------- ----------------
CDB1      READ WRITE           CDB1
 
NAME      OPEN_MODE            INSTANCE_NAME
--------- -------------------- ----------------
CDB1      READ WRITE           CDB1
 
[oracle@VM122 ~]$ for i in {1..3} ; do sqlplus -L -s sys/oracle@//localhost/pdb1/CDB2 as sysdba <<< 'select name,open_mode,instance_name from v$instance , v$database;'; done
 
NAME      OPEN_MODE            INSTANCE_NAME
--------- -------------------- ----------------
CDB1      READ ONLY WITH APPLY CDB2
 
NAME      OPEN_MODE            INSTANCE_NAME
--------- -------------------- ----------------
CDB1      READ ONLY WITH APPLY CDB2
 
NAME      OPEN_MODE            INSTANCE_NAME
--------- -------------------- ----------------
CDB1      READ ONLY WITH APPLY CDB2

Of course this is not what you want. And we must not start or stop the default services. For the application, the best you can do is to create your service. And if you want to be able to connect to the Active Data Guard standby, which is opened in read-only, then you can create a ‘read-write’ service and a ‘read-only’ service that you start depending on the role.

Create and Start a read-write service on the primary

This example supposes that you have only Oracle Database software installed. If you are in RAC, with the resources managed by Grid Infrastructure, or simply with Oracle Restart, creating a service is easy with srvctl, and you add it to a PDB with ‘-pdb’ and also with a role to start it automatically in the primary or in the standby. But without it, you use dbms_service:

SQL> connect /@CDB1 as sysdba
Connected.
 
SQL> alter session set container=pdb1;
Session altered.
 
SQL> exec dbms_service.create_service(service_name=>'pdb1_RW',network_name=>'pdb1_RW');
PL/SQL procedure successfully completed.
 
SQL> exec dbms_service.start_service(service_name=>'pdb1_RW');
PL/SQL procedure successfully completed.
 
SQL> alter session set container=cdb$root;
Session altered.

The service is created, stored in SERVICE$ visible with DBA_SERVICES:

SQL> select name,name_hash,network_name,creation_date,pdb from cdb_services order by con_id,service_id;
NAME         NAME_HASH NETWORK_NAME   CREATION_DATE   PDB
----         --------- ------------   -------------   ---
pdb1_RW     3128030313 pdb1_RW        03-FEB-18       PDB1
pdb1        1888881990 pdb1           11-JAN-18       PDB1

Save state

I have created and started the PDB1_RW service. However, if I restart the database, the service will not start automatically. How do you ensure that the PDB1 pluggable database starts automatically when you open the CDB? You ‘save state’ when it is opened. It is the same for the services you create. You need to ‘save state’ when they are opened.

SQL> alter pluggable database all save state;
Pluggable database ALL altered.

The information is stored in PDB_SVC_STATE$, and I’m not aware of a dictionary view on it:

SQL> select name,name_hash,network_name,creation_date,con_id from v$active_services order by con_id,service_id;
 
NAME         NAME_HASH NETWORK_NAME   CREATION_DATE     CON_ID
----         --------- ------------   -------------     ------
pdb1_RW     3128030313 pdb1_RW        03-FEB-18              4
pdb1        1888881990 pdb1           11-JAN-18              4
 
SQL> select * from containers(pdb_svc_state$);
 
  INST_ID INST_NAME   PDB_GUID                                PDB_UID     SVC_HASH   SPARE1   SPARE2   SPARE3   SPARE4 SPARE5   SPARE6     CON_ID 
  ------- ---------   --------                                -------     --------   ------   ------   ------   ------ ------   ------     ------
        1 CDB1        627F7512A0452FD4E0537A38A8C055C0     2872139986   3128030313                                                              1 

The name is not in this table, you have to join with v$services using(name_hash):

SQL> select name,name_hash,network_name,creation_date,con_id from v$active_services order by con_id,service_id;
 
NAME                NAME_HASH NETWORK_NAME   CREATION_DATE     CON_ID 
----                --------- ------------   -------------     ------
SYS$BACKGROUND      165959219                26-JAN-17              1 
SYS$USERS          3427055676                26-JAN-17              1 
CDB1_CFG           1053205690 CDB1_CFG       24-JAN-18              1 
CDB1_DGB            184049617 CDB1_DGB       24-JAN-18              1 
CDB1XDB            1202503288 CDB1XDB        11-JAN-18              1 
CDB1               1837598021 CDB1           11-JAN-18              1 
pdb1               1888881990 pdb1           11-JAN-18              4 
pdb1_RW            3128030313 pdb1_RW        03-FEB-18              4 

So, in addition to storing the PDB state in PDBSTATE$, visible with dba_pdb_saved_states, the service state is also stored. Note that they are at different level. PDBSTATE$ is a data link: stored on CDB$ROOT only (because the data must be read before opening the PDB) but PDB_SVC_STATE$ is a local table in the PDB as the services can be started only when the PDB is opened.

This new service is immediately registered on CDB1:

Service "pdb1" has 2 instance(s).
  Instance "CDB1", status READY, has 1 handler(s) for this service...
  Instance "CDB2", status READY, has 1 handler(s) for this service...
Service "pdb1_RW" has 1 instance(s).
  Instance "CDB1", status READY, has 1 handler(s) for this service...
The command completed successfully

Create and Start a read-only service for the standby

If you try to do the same on the standby for a PDB1_RO service, you cannot because service information has to be stored in the dictionary:

SQL> exec dbms_service.create_service(service_name=>'pdb1_RO',network_name=>'pdb1_RO');
 
Error starting at line : 56 File @ /media/sf_share/122/blogs/pdb_svc_standby.sql
In command -
BEGIN dbms_service.create_service(service_name=>'pdb1_RO',network_name=>'pdb1_RO'); END;
Error report -
ORA-16000: database or pluggable database open for read-only access

So, the read-only service has to be created on the primary:

SQL> connect /@CDB1 as sysdba
Connected.
SQL> alter session set container=pdb1;
Session altered.
 
SQL> exec dbms_service.create_service(service_name=>'pdb1_RO',network_name=>'pdb1_RO');
 
SQL> select name,name_hash,network_name,creation_date,pdb from cdb_services order by con_id,service_id;
NAME         NAME_HASH NETWORK_NAME   CREATION_DATE   PDB
----         --------- ------------   -------------   ---
pdb1_RW     3128030313 pdb1_RW        03-FEB-18       PDB1
pdb1_RO     1562179816 pdb1_RO        03-FEB-18       PDB1
pdb1        1888881990 pdb1           11-JAN-18       PDB1

The SERVICE$ dictionary table is replicated to the standby, so I can I can start it on the standby:

SQL> connect /@CDB2 as sysdba
Connected.
SQL> alter session set container=pdb1;
Session altered.
 
SQL> exec dbms_service.start_service(service_name=>'pdb1_RO');
PL/SQL procedure successfully completed.

Here is what is registered to the listener:

Service "pdb1" has 2 instance(s).
  Instance "CDB1", status READY, has 1 handler(s) for this service...
  Instance "CDB2", status READY, has 1 handler(s) for this service...
Service "pdb1_RO" has 1 instance(s).
  Instance "CDB2", status READY, has 1 handler(s) for this service...
Service "pdb1_RW" has 1 instance(s).
  Instance "CDB1", status READY, has 1 handler(s) for this service...
The command completed successfully

Now, the PDB_RO connects to the standby and PDB_RW to the primary. Perfect.

No ‘save state’ on the standby

At this point, you would like to have the PDB_RO started when PDB1 is opened on the standby, but ‘save state’ is impossible on a read-only database:

SQL> alter session set container=cdb$root;
Session altered.
 
SQL> alter pluggable database all save state;
 
Error starting at line : 84 File @ /media/sf_share/122/blogs/pdb_svc_standby.sql
In command -
alter pluggable database all save state
Error report -
ORA-16000: database or pluggable database open for read-only access

You can’t manage the state (open the PDB, start the services) in the standby database.

The primary ‘save state’ is replicated in standby

For the moment, everything is ok with my services:

Service "pdb1_RO" has 1 instance(s).
  Instance "CDB2", status READY, has 1 handler(s) for this service...
Service "pdb1_RW" has 1 instance(s).
  Instance "CDB1", status READY, has 1 handler(s) for this service...

If I restart the primary CDB1, everything is ok again because I saved the state of the PDB and the service. But what happens when the standby CDB2 restarts?

SQL> connect /@CDB2 as sysdba
Connected.
SQL> startup force;
...
SQL> show pdbs
 
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         4 PDB1                           MOUNTED

The PDB is not opened: the ‘saved state’ for PDB is not read in the standby.
However, when I open the PDB, it seems that the ‘saved state’ for service is applied, and this one is replicated from the primary:

SQL> alter pluggable database PDB1 open;
Pluggable database altered.
SQL> host lsnrctl status
...
Service "pdb1" has 2 instance(s).
  Instance "CDB1", status READY, has 1 handler(s) for this service...
  Instance "CDB2", status READY, has 1 handler(s) for this service...
Service "pdb1_RW" has 2 instance(s).
  Instance "CDB1", status READY, has 1 handler(s) for this service...
  Instance "CDB2", status READY, has 1 handler(s) for this service...
The command completed successfully

My PDB1_RW is registered for both, connections will connect at random to the primary or the standby, and then the transactions will fail half of the times. It will be the same in case of switchover. This is not correct.

Save state instances=()

What I would like is the possibility to save state for a specific DB_UNIQUE_NAME, like with pluggable ‘spfile’ parameters. But this is not possible. What is possible is to mention an instance but you can use it only for the primary instance where you save the state (or you get ORA-65110: Invalid instance name specified) and anyway, this will not be correct after a switchover.

So what?

Be careful, with services and ensure that the services used by the application are registered only for the correct instance. Be sure that this persists when the instances are restarted. For this you must link a service name to a database role. This cannot be done correctly with ‘save state’. You can use startup triggers, or better, Grid Infrastructure service resources.

Do not connect to the default service with the PDB name, you cannot remove it and cannot stop it, so you may have the same name for different instances in a Data Guard configuration. You can register the standby instances to different local listeners, to avoid the confusion, but you may still register to the same SCAN listener.

Create your own services, start them depending on the database role, and do not use ‘save state’ in a physical standby configuration.

L’article Multitenant, PDB, ‘save state’, services and standby databases est apparu en premier sur dbi Blog.

.
In the previous post https://www.dbi-services.com/blog/12c-multitenant-internals-pdb-replay-ddl-for-common-users I’ve done some DDL on a common user to show how this is replayed later for PDBs that were not opened at that time. But what happens when one of the DDL fails on one PDB?

PDB$LASTREPLAY

In the last post, the C##USER1 common user was created and all pluggable databases (PDB1 with con_id=3 and PDB2 with con_id=4) were opened and synchronized:

SQL> select con_id,scnwrp,scnbas,ctime,name,auxname1,auxname2,opcode,replay#,sqlstmt from containers(pdb_sync$) where opcode=-1 and bitand(flags,8)!=8 order by con_id,replay#;
 
  CON_ID   SCNWRP   SCNBAS CTIME                NAME             AUXNAME1   AUXNAME2     OPCODE   REPLAY# SQLSTMT
  ------   ------   ------ -----                ----             --------   --------     ------   ------- -------
       1        0        0 26-jan-17 01:53:02   PDB$LASTREPLAY                               -1        10
       3        0        0 26-jan-17 01:53:02   PDB$LASTREPLAY                               -1        10
       4        0        0 26-jan-17 01:53:02   PDB$LASTREPLAY                               -1        10 

When REPLAY# in the PDB is equal to the CDB$ROOT one, this means that there are no additional statements to replicate on the PDB.

I have PDB1 opened read write and PDB2 in read only:

SQL> show pdbs
 
  CON_ID CON_NAME   OPEN MODE    RESTRICTED
  ------ --------   ---- ----    ----------
       2 PDB$SEED   READ ONLY    NO
       3 PDB1       READ WRITE   NO
       4 PDB2       READ ONLY    NO

For the demo my user’s default tablespace is SYSTEM:

SQL> select con_id,username,common,default_tablespace from cdb_users where username='C##USER1' order by 1;
 
  CON_ID USERNAME   COMMON   DEFAULT_TABLESPACE
  ------ --------   ------   ------------------
       1 C##USER1   YES      SYSTEM
       3 C##USER1   YES      SYSTEM
       4 C##USER1   YES      SYSTEM

Failure in opened containers

I want to change the default tablespace for C##USER1 and I have a USERS tablespace in CDB$ROOT (but not in the PDBs):

SQL> alter user C##USER1 default tablespace USERS;
 
Error starting at line : 50 File @ common-users-pdb-sync.sql
In command -
alter user C##USER1 default tablespace USERS
Error report -
ORA-65048: error encountered when processing the current DDL statement in pluggable database PDB1
ORA-00959: tablespace 'USERS' does not exist

As we have seen in the last post, the DDL is executed on all containers that are opened read write. Here it is fine on CDB$ROOT but fails on PDB1.

Then I create the USERS tablespace in PDB1:

SQL> alter session set container=PDB1;
Session altered.
SQL> create tablespace USERS datafile '/u01/oradata/CDB2/PDB1/users.dbf' size 5M;
Tablespace USERS created.
SQL> alter session set container=CDB$ROOT;
Session altered.

And now, the statement is successful in CDB$ROOT, replicated on PDB1:

SQL> alter user C##USER1 default tablespace USERS;
User C##USER1 altered.

This is nice: the statement is successful in all containers or fails. When it is successful, statements are recorded in PDB_SYNC$:

SQL> select con_id,scnwrp,scnbas,ctime,name,auxname1,auxname2,opcode,replay#,sqlstmt from containers(pdb_sync$) where con_id=1 and bitand(flags,8)!=8 order by con_id,replay#;
 
  CON_ID   SCNWRP    SCNBAS CTIME                NAME             AUXNAME1   AUXNAME2     OPCODE   REPLAY# SQLSTMT
  ------   ------    ------ -----                ----             --------   --------     ------   ------- -------
       1        0         0 26-jan-17 01:53:02   PDB$LASTREPLAY                               -1        11
       1        0    852610 26-jan-17 02:57:26   CTXSYS                      SYS               5         1 alter user CTXSYS account unlock identified by *^@
       1        0    853177 26-jan-17 02:57:34   CTXSYS                      SYS               5         2 alter user CTXSYS password expire account lock^@
       1        0   1405359 26-jan-17 03:31:31   SYSTEM                      SYS               5         3 alter user system account lock password expire^@
       1        0   1408693 23-dec-17 11:34:43   SYS                         SYS               5         4 alter user sys account unlock identified by *^@
       1        0   1408703 23-dec-17 11:34:43   SYSTEM                      SYS               5         5 alter user system account unlock identified by *^@
       1        0   1466615 29-dec-17 09:26:56   C##PROFILE1                 SYS               7         6 create profile C##PROFILE1 limit inactive_account_time 15 container=all^@
       1        0   1466641 29-dec-17 09:26:57   C##ROLE1                    SYS               3         7 create role C##ROLE1 container=all^@
       1        0   1466748 29-dec-17 09:26:58   C##USER1                    SYS               1         8 create user C##USER1 identified by * container=all^@
       1        0   1466812 29-dec-17 09:26:59   C##USER1                    SYS               5         9 alter user C##USER1 profile C##PROFILE1^@
       1        0   1466853 29-dec-17 09:26:59   C##USER1         C##ROLE1   SYS              10        10 grant C##ROLE1 to C##USER1 container=all^@
       1        0   1467010 29-dec-17 09:27:01   C##USER1                    SYS               5        11 alter user C##USER1 default tablespace USERS^@

Failure in replay at open for closed containers

But PDB2 is not synchronized because it was not opened read write:

SQL> select con_id,scnwrp,scnbas,ctime,name,auxname1,auxname2,opcode,replay#,sqlstmt from containers(pdb_sync$) where con_id>1 and bitand(flags,8)!=8 order by con_id,replay#;
 
  CON_ID   SCNWRP   SCNBAS CTIME                NAME             AUXNAME1   AUXNAME2     OPCODE   REPLAY# SQLSTMT
  ------   ------   ------ -----                ----             --------   --------     ------   ------- -------
       3        0        0 26-jan-17 01:53:02   PDB$LASTREPLAY                               -1        11
       4        0        0 26-jan-17 01:53:02   PDB$LASTREPLAY                               -1        10

But I don’t have a USERS tablespace in PDB2, so the replay will fail:

SQL> alter pluggable database PDB2 close;
Pluggable database PDB2 altered.
 
SQL> alter pluggable database PDB2 open;
ORA-24344: success with compilation error
Pluggable database PDB2 altered.

This is a warning only. The SQlcl feedback is a bit misleading, mentioning a compilation error because this is where we used to have warnings, but the SQl*Plus message is more clear:

SQL> alter pluggable database PDB2 open;
Warning: PDB altered with errors.

The PDB2 cannot be left closed, because you need to create a tablespace here. But it cannot be opened to everyone, because it is not in sync with CDB$ROOT. So what happens is that the PDB is opened in restricted mode:

SQL> show pdbs
 
  CON_ID CON_NAME   OPEN MODE    RESTRICTED
  ------ --------   ---- ----    ----------
       2 PDB$SEED   READ ONLY    NO
       3 PDB1       READ WRITE   NO
       4 PDB2       READ WRITE   YES

Note that if you look at PDB_SYNC$ in the PDB at that time, it looks like REPLAY#=11 has increased but you also see rows for the statement that has to be run. You have to connect to the PDB because containers() do not run in restricted session containers:

SQL> alter session set container=PDB2;
Session altered.
 
SQL> select con_id,scnwrp,scnbas,ctime,name,auxname1,auxname2,opcode,replay#,sqlstmt from containers(pdb_sync$) where con_id>1 and bitand(flags,8)!=8 order by con_id,replay#;
 
  CON_ID   SCNWRP    SCNBAS CTIME                NAME             AUXNAME1   AUXNAME2     OPCODE   REPLAY# SQLSTMT
  ------   ------    ------ -----                ----             --------   --------     ------   ------- -------
       4        0         0 26-jan-17 01:53:02   PDB$LASTREPLAY                               -1        11
       4        0   1469022 29-dec-17 09:27:02   C##USER1                    SYS               5        11 alter user C##USER1 default tablespace USERS^@
 
SQL> alter session set container=CDB$ROOT;
Session altered.

Actually, the attempt to sync has inserted the statements and pushed the last replay indicator. Now, the PDB has all information to do a sync without the need to go to CDB$ROOT. The DDL was not replayed, but has been stored locally. When the sync will be successful, statements will be removed from the local PDB_SYNC$ leaving only the LASTREPLAY indicator.

PDB_PLUG_IN_VIOLATIONS

More info about the warning is stored in PDB_ALERT$ which you query from PDB_PLUG_IN_VIOLATIONS (the strange name reminds the TRANSPORT_SET_VIOLATIONS view used by DBMS_TTS):

SQL> select name, cause, type, status,action,message,time from pdb_plug_in_violations;
 
NAME   CAUSE          TYPE    STATUS    ACTION   MESSAGE                                                                               TIME
----   -----          ----    ------    ------   -------                                                                               -------------------------------
PDB2   Sync Failure   ERROR   PENDING            Sync PDB failed with ORA-959 during 'alter user C##USER1 default tablespace USERS'    29-DEC-17 09.27.03.266780000 PM

Here you have the statement that failed and the error number, but no recommended ACTION. However, ORA-959 is “tablespace ‘%s’ does not exist” which gives a clue about the problem encountered.

As the PDB is opened a DBA (with RESTRICTED SESSION privilege) can add the tablespace:

SQL> alter session set container=PDB2;
Session altered.
 
SQL> create tablespace USERS datafile '/u01/oradata/CDB2/PDB2/users.dbf' size 5M;
Tablespace USERS created.

But you cannot simply disable restricted session:

SQL> alter system disable restricted session;
SQL> alter system disable restricted session;
 
Error starting at line : 74 File @ common-users-pdb-sync.sql
In command -
alter system disable restricted session
Error report -
ORA-65144: ALTER SYSTEM DISABLE RESTRICTED SESSION is not permitted
65144. 00000 -  "ALTER SYSTEM DISABLE RESTRICTED SESSION is not permitted"
*Cause:    An attempt was made to disable a restricted session while an unresolved error existed in PDB_PLUG_IN_VIOLATIONS.
*Action:   Resolve all of the errors before trying to disable a restricted session.

One solution is to close and open the PDB to get the DDL replay:

SQL> alter session set container=CDB$ROOT;
Session altered.
SQL> alter pluggable database PDB2 close;
Pluggable database PDB2 altered.
SQL> alter pluggable database PDB2 open;
Pluggable database PDB2 altered.

The other solution is to call DBMS_PDB.SYNC_PDB and disable restricted mode:

SQL> exec dbms_pdb.sync_pdb; commit;
PL/SQL procedure successfully completed.
 
SQL> alter system disable restricted session;
System DISABLE altered.

In both case, no warning here, and no restricted mode:

SQL> show pdbs
 
  CON_ID CON_NAME   OPEN MODE    RESTRICTED
  ------ --------   ---- ----    ----------
       2 PDB$SEED   READ ONLY    NO
       3 PDB1       READ WRITE   NO
       4 PDB2       READ WRITE   NO

and the PDB_PLUG_IN_VIOLATIONS is updated to flag the issue as resolved:

SQL> select name, cause, type, status,action,message,time from pdb_plug_in_violations;
 
NAME   CAUSE          TYPE    STATUS     ACTION   MESSAGE                                                                               TIME
----   -----          ----    ------     ------   -------                                                                               -------------------------------
PDB2   Sync Failure   ERROR   RESOLVED            Sync PDB failed with ORA-959 during 'alter user C##USER1 default tablespace USERS'    29-DEC-17 09.27.04.093659000 PM

At that time, the local PDB_SYNC$ table in PDB2 contains only the PDB$LASTREPLAY row, with the same value as in the CDB$ROOT table. The rows with the statements have been deleted once the DDL has been successfully replayed:

SQL> select con_id,scnwrp,scnbas,ctime,name,auxname1,auxname2,opcode,replay#,sqlstmt from containers(pdb_sync$) where con_id>1 and bitand(flags,8)!=8 order by con_id,replay#;
 
  CON_ID   SCNWRP    SCNBAS CTIME                NAME             AUXNAME1   AUXNAME2     OPCODE   REPLAY# SQLSTMT
  ------   ------    ------ -----                ----             --------   --------     ------   ------- -------
       4        0         0 26-jan-17 01:53:02   PDB$LASTREPLAY                               -1        11

So what?

The mechanism is simple: record what is done in CDB$ROOT, replicate it in PDBs when possible (opened read-write) and try to replay it, mark the last replay step. For containers that were not writeable, at open, the DDL is replicated on the PDBs that lag being CDB$ROOT and replay step is updated. Then the DDL is replayed. When sucessful, the statement is removed from the replicated DDL. When it fails, you get a warning, and a message in PDB_PLUG_IN_VIOLATIONS, and the PDB is opened in restricted session mode to let you solve the problem.
If you can fix the issue so that the DDL to be replayed is successful, then you can just sync and disable restricted session, or simply close and re-open the PDB.
If you can’t fix it I suppose you need to hack the statements in the local PDB_SYNC$, with Oracle Support agreement of course, and make sure that you arrive to a state which is consistent with the other containers, especially CDB$ROOT.

L’article 12c Multitenant internals: PDB_PLUG_IN_VIOLATIONS est apparu en premier sur dbi Blog.

.
In multitenant, you can create common Users, Roles, and Profiles. You create them in CDB$ROOT, with the CONTAINER=ALL clause (which is optional because it is the only possible value when connected to CDB$ROOT) but they are visible to all containers. As the goal of multitenant is to avoid to duplicate common metadata to all containers, You may think that they are visible through those magic metadata links. But that’s actually wrong: they are simply replicated with a very simple mechanism: the DDL for common objects is replayed into each user PDB.

I’m connected to CDB2’s CDB$ROOT and I have two pluggable databases:

SQL> show pdbs
 
  CON_ID CON_NAME   OPEN MODE    RESTRICTED
  ------ --------   ---- ----    ----------
       2 PDB$SEED   READ ONLY    NO
       3 PDB1       READ WRITE   NO
       4 PDB2       MOUNTED

PDB1 is opened and PDB2 is closed.

PDB_SYNC$

In this example, I’ll query PDB_SYNC$ which is the table where Oracle stores all DDL for common users, roles, or profiles in order to be able to replay it later:

SQL> select con_id,scnwrp,scnbas,ctime,name,auxname1,auxname2,opcode,replay#,sqlstmt from containers(pdb_sync$) where con_id=1 and bitand(flags,8)!=8 order by con_id,replay#;
 
  CON_ID   SCNWRP    SCNBAS CTIME                NAME             AUXNAME1   AUXNAME2     OPCODE   REPLAY# SQLSTMT
  ------   ------    ------ -----                ----             --------   --------     ------   ------- -------
       1        0    852610 26-jan-17 02:57:26   CTXSYS                      SYS               5         1 alter user CTXSYS account unlock identified by *^@
       1        0    853177 26-jan-17 02:57:34   CTXSYS                      SYS               5         2 alter user CTXSYS password expire account lock^@
       1        0   1405359 26-jan-17 03:31:31   SYSTEM                      SYS               5         3 alter user system account lock password expire^@
       1        0   1408693 23-dec-17 11:34:43   SYS                         SYS               5         4 alter user sys account unlock identified by *^@
       1        0   1408703 23-dec-17 11:34:43   SYSTEM                      SYS               5         5 alter user system account unlock identified by *^@
       1        0         0 26-jan-17 01:53:02   PDB$LASTREPLAY                               -1         5

I excluded the bitand(flags,8)=8 because it concerns application containers. I query with the container() to show the con_id but this is for con_id=1 which is the CDB$ROOT.

You can see some DDL for CTXSYS recorded on January 26th which is the day where this release (12.2.0.1) was built. I used a template with datafiles to create the CDB with DBCA. And you see some DDL to unlock SYS and SYSTEM on December 23rd when I created the database. You can also see that they are ordeded in sequence with REPLAY#.

More interesting is the OPCODE=-1 which is PDB$LASTREPLAY and looks like the last value of REPLAY#. This means that on this container, CDB$ROOT, all statements where REPLAY#<=5 was run.

With a similar query, I query the opened PDBs:

SQL> select con_id,scnwrp,scnbas,ctime,name,auxname1,auxname2,opcode,replay#,sqlstmt from containers(pdb_sync$) where con_id>1 and bitand(flags,8)!=8 order by con_id,replay#;
 
  CON_ID   SCNWRP   SCNBAS CTIME                NAME             AUXNAME1   AUXNAME2     OPCODE   REPLAY# SQLSTMT
  ------   ------   ------ -----                ----             --------   --------     ------   ------- -------
       3        0        0 26-jan-17 01:53:02   PDB$LASTREPLAY                               -1         5

There only one row here in CON_ID=3, which is PDB1: the PDB$LASTREPLAY mentioning that all statements up to REPLAY=5 have been run also in this container.
I don’t see PDB2 (CON_ID=4) here because the container() clause cannot query closed containers.

CONTAINER=ALL DDL

I’ll run some common DLL to create a profile, a role and a user:

SQL> create profile C##PROFILE1 limit inactive_account_time 15 container=all;
Profile C##PROFILE1 created.
 
SQL> create role C##ROLE1 container=all;
Role C##ROLE1 created.
 
SQL> create user C##USER1 identified by oracle container=all;
User C##USER1 created.
 
SQL> alter user C##USER1 profile C##PROFILE1;
User C##USER1 altered.
 
SQL> grant C##ROLE1 to C##USER1 container=all;
Grant succeeded.

The C## prefix is mandatory to isolate the common user namespace. You can change it with the common_prefix parameter. You can even set it to the empty string, but then you have a risk of namespace collision when you plug a PDB between CDB having different common profiles or roles.
The CONTAINER=ALL is the default and the only possibility when connected to CDB$ROOT so it is optional. I recommend to mention it explicitly in order to avoid problems when running the same DDL in CDB$ROOT and in PDBs.

All those DDL have been recorded into PDB_SYNC$ and the REPLAY# has been increased:

SQL> select con_id,scnwrp,scnbas,ctime,name,auxname1,auxname2,opcode,replay#,sqlstmt from containers(pdb_sync$) where con_id=1 and bitand(flags,8)!=8 order by con_id,replay#;
 
  CON_ID   SCNWRP    SCNBAS CTIME                NAME             AUXNAME1   AUXNAME2     OPCODE   REPLAY# SQLSTMT
  ------   ------    ------ -----                ----             --------   --------     ------   ------- -------
       1        0    852610 26-jan-17 02:57:26   CTXSYS                      SYS               5         1 alter user CTXSYS account unlock identified by *^@
       1        0    853177 26-jan-17 02:57:34   CTXSYS                      SYS               5         2 alter user CTXSYS password expire account lock^@
       1        0   1405359 26-jan-17 03:31:31   SYSTEM                      SYS               5         3 alter user system account lock password expire^@
       1        0   1408693 23-dec-17 11:34:43   SYS                         SYS               5         4 alter user sys account unlock identified by *^@
       1        0   1408703 23-dec-17 11:34:43   SYSTEM                      SYS               5         5 alter user system account unlock identified by *^@
       1        0   1466615 29-dec-17 09:26:56   C##PROFILE1                 SYS               7         6 create profile C##PROFILE1 limit inactive_account_time 15 container=all^@
       1        0   1466641 29-dec-17 09:26:57   C##ROLE1                    SYS               3         7 create role C##ROLE1 container=all^@
       1        0   1466748 29-dec-17 09:26:58   C##USER1                    SYS               1         8 create user C##USER1 identified by * container=all^@
       1        0   1466812 29-dec-17 09:26:59   C##USER1                    SYS               5         9 alter user C##USER1 profile C##PROFILE1^@
       1        0   1466853 29-dec-17 09:26:59   C##USER1         C##ROLE1   SYS              10        10 grant C##ROLE1 to C##USER1 container=all^@
       1        0         0 26-jan-17 01:53:02   PDB$LASTREPLAY                               -1        10

PDB1 (CON_ID=3) was opened read write, and then has been synchronized (the DDL has been run in the container to create the same profile, role and user) and the PDB$LASTREPLAY has been updated in this container to show that all has been done:

SQL> select con_id,scnwrp,scnbas,ctime,name,auxname1,auxname2,opcode,replay#,sqlstmt from containers(pdb_sync$) where con_id>1 and bitand(flags,8)!=8 order by con_id,opcode,replay#;
%nbsp;
  CON_ID   SCNWRP   SCNBAS CTIME                NAME             AUXNAME1   AUXNAME2     OPCODE   REPLAY# SQLSTMT
  ------   ------   ------ -----                ----             --------   --------     ------   ------- -------
       3        0        0 26-jan-17 01:53:02   PDB$LASTREPLAY                               -1        10

MOUNTED or READ ONLY

I open the PDB2 read only because I want to see what is in PDB_SYNC$ there. But READ ONLY means that the DDL cannot be run because no write is allowed in the local dictionary.

SQL> alter pluggable database PDB2 open read only;
Pluggable database PDB2 altered.

Running the same query as above, I can see that PDB2 (CON_ID=4) is synchronized only up to the statements with REPLAY#=5 because my DDL was not replicated there.

SQL> select con_id,scnwrp,scnbas,ctime,name,auxname1,auxname2,opcode,replay#,sqlstmt from containers(pdb_sync$) where con_id>1 and bitand(flags,8)!=8 order by con_id,opcode,replay#;
 
  CON_ID   SCNWRP   SCNBAS CTIME                NAME             AUXNAME1   AUXNAME2     OPCODE   REPLAY# SQLSTMT
  ------   ------   ------ -----                ----             --------   --------     ------   ------- -------
       3        0        0 26-jan-17 01:53:02   PDB$LASTREPLAY                               -1        10
       4        0        0 26-jan-17 01:53:02   PDB$LASTREPLAY                               -1         5

Sync at OPEN

When I open PDB2 in read write mode, the DDL can be synchronized:

SQL> alter pluggable database PDB2 open read write force;
Pluggable database PDB2 altered.

At open, the DDL from REPLAY#>5 has been replayed and once opened the PDB is in sync with CDB$ROOT:

SQL> select con_id,scnwrp,scnbas,ctime,name,auxname1,auxname2,opcode,replay#,sqlstmt from containers(pdb_sync$) where con_id>1 and bitand(flags,8)!=8 order by con_id,opcode,replay#;
 
  CON_ID   SCNWRP   SCNBAS CTIME                NAME             AUXNAME1   AUXNAME2     OPCODE   REPLAY# SQLSTMT
  ------   ------   ------ -----                ----             --------   --------     ------   ------- -------
       3        0        0 26-jan-17 01:53:02   PDB$LASTREPLAY                               -1        10
       4        0        0 26-jan-17 01:53:02   PDB$LASTREPLAY                               -1        10

So what?

The common users, roles and profiles are not stored only in CDB$ROOT to be shared, but rather replicated to all PDBs. The DDL is replicated synchronously to all opened pluggable databases in read write, and stored into the CDB$ROOT PDB_SYNC$ table to be replayed later when non-synced PDBs are opened. I’ll show in the next post what happens when the DDL is in error.

Note that even when all pluggable databases are opened read write, the DDL is stored and they are purged later (when replayed on all PDBs) because they are needed when you create a new PDB and open it. The PDB$SEED has REPLAY#=0 for PDB$LASTREPLAY which means that all statements will be replayed.

This is 12.2.0.1 where this mechanism is only for common users, roles and profiles having DDL in CDB$ROOT. With application containers, more than that is recorded: all DML and DDL run between the ‘begin install/upgrade/patch’ and ‘end install/upgrade/patch’ in the application root. Then, the statements can be replayed into the application PDB with a simple SYNC command. In the future release (18c) we expect to have that application root behavior ported to CDB$ROOT so that we don’t have to run catupgrd.sql in all containers. Then the PDB will probably be patched or upgraded when opened.

L’article 12c Multitenant internals: PDB replay DDL for common users est apparu en premier sur dbi Blog.

.
When I explain the multitenant internals, I show that all metadata about system procedures and packages are stored only in CDB$ROOT and are accessed from the PDBs through metadata links. I take an example with DBMS_SYSTEM that has nothing in SOURCE$ of the PDB. But I show that we can compile it from the PDB. This is my way to prove that the session can access the system objects, internally switching the session to the root container when it needs to read SOURCE$. At DOAG Conference I had a very interesting question about what happens exactly in CDB$ROOT: Is the session really executing all the DML on the internal tables storing the compiled code of the procedure?

My first answer was something like ‘why not’ because the session in a PDB can switch and do modifications into CDB$ROOT internally. For example, even a local PDB DBA can change some ‘spfile’ parameters which are actually stored in the CDB$ROOT. But then I realized that the question goes further: is the PDB session really compiling the DBMS_SYSTEM package in the CDB$ROOT? Actually, there are some DDL that are transformed to ‘no-operation’ when executed on the PDB.

To see which ones are concerned, the best is to trace:

SQL> alter session set events='10046 trace name context forever, level 4';
Session altered.
SQL> alter session set container=PDB1;                                                                              
Session altered.              
SQL> alter package dbms_system compile;                                                                               
Package altered.           
SQL> alter session set events='10046 trace name context off';
Session altered.

I’ll not show the whole trace here. For sure I can see that the session switches to CDB$ROOT to read the source code of the package:

*** 2017-11-22T08:36:01.963680+01:00 (CDB$ROOT(1))
=====================
PARSING IN CURSOR #140650193204552 len=54 dep=1 uid=0 oct=3 lid=0 tim=5178881528 hv=696375357 ad='7bafeab8' sqlid='9gq78x8ns3q1x'
select source from source$ where obj#=:1 order by line
END OF STMT
PARSE #140650193204552:c=0,e=290,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=5178881527
EXEC #140650295606992:c=1000,e=287,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=4,plh=813480514,tim=5178881999
FETCH #140650295606992:c=0,e=35,p=0,cr=4,cu=0,mis=0,r=1,dep=2,og=4,plh=813480514,tim=5178882057
CLOSE #140650295606992:c=0,e=12,dep=2,type=3,tim=5178882104

That was my point about metadata links. But now about modifications.

As I need to see only the statements, I can use TKPROF to get them aggregated, but then the container switch – like (CDB$ROOT(1)) here – is ignored.

Here is a small AWK script I use to add the Container ID to the SQL ID so that it is visible and detailed into TKPROF output:

awk '/^[*]{3}/{con=$3}/^PARSING IN/{sub(/sqlid=./,"&"con" ")}{print > "con_"FILENAME }'

Then I run TKPROF on the resulting file, with ‘sort=(execu)’ so that I have the modifications (insert/delete/update) first. The result starts with something like this:

SQL ID: (PDB1(3)) 1gfaj4z5hn1kf Plan Hash: 1110520934
 
delete from dependency$
where
 d_obj#=:1

I know that dependencies are replicated into all containers (because table metadata is replicated into all containers) so I see following tables modified in the PDB: DEPENDENCY$, ACCESS$, DIANA_VERSION$, and of course OBJ$.

But to answer the initial question, there are no modifications done in the CDB$ROOT. Only SELECT statements there, on SOURCE$, SETTINGS$, CODEAUTH$, WARNING_SETTINGS$

So, probably, the updates have been transformed to no-op operations once the session is aware that the source is the same (same signature) and it just reads the compilation status.

Just as a comparison, tracing the same compilation when done on the CDB$ROOT will show inserts/delete/update on ARGUMENT$, PROCEDUREINFO$, SETTINGS$, PROCEDUREPLSQL$, IDL_UB1$, IDL_SB4$, IDL_UB2$, IDL_CHAR$, … all those tables sorting the compiled code.

So basically, when running DDL on metadata links in a PDB, not all the work is done in the CDB, especially not writing again what is already there (because you always upgrade the CDB$ROOT first). However, up to 12.2 we don’t see a big difference in time. This should change in 18c where the set of DDL to be run on the PDB will be pre-processed to avoid unnecessary operations.

L’article 12c Multitenant Internals: compiling system package from PDB est apparu en premier sur dbi Blog.

.
This month, I’ll talk – with lot of demos – about multitenant internals at DOAG conference.
The multitenant dictionary architecture starts with a simple idea: system metadata and data are in CDB$ROOT and user metadata and data are in PDB. And it could have been that simple. When a session connected to a PDB needs to read some system information, the session context is switched to the CDB$ROOT container and reads from CDB$ROOT SYS tablespace and objects, and then switches back to the PDB container. This is implemented by metadata and data links: the PDB lists the objects for which the session has to switch to CDB$ROOT to get metadata or data.

But, for compatibility reason, and ease of administration, the dictionary views must display information from both containers, transparently, and then things become a little more complex with common views and extended data views.

At Oracle Open World, the multitenant architects, in the #PDBExpert session, answered questions about the multitenant architecture posted on Twitter. My first question (because I was investigating a bug at that time) was about some views, such as INT$INT$DBA_CONSTRAINTS, introduced to implement the complexity of showing the same information in dictionary views as the ones we had on non-CDB. Of course, the architects didn’t want to go too far on this and had a very accurate answer: INT$ is for internal, and here you have two ‘INT$’ so you shouldn’t look at that.

But I like to understand how things work and here is the explanation of these INT$INT$ views. And I’m not even sure that INT is for ‘internal’ but maybe ‘intermediate’. But for sure, the $ at the end is used by Oracle internal dictionary objects.

INT$ Extended Data views

We are used to seeing all objects, system ones and user ones, listed by the dictionary views. For example, DBA_PROCEDURES shows all procedures, system and user ones, and then have to read from both containers (current PDB and CDB$ROOT) through extended data links. ALL_PROCEDURES shows all procedures accessible by the user, and they also have to switch to CDB$ROOT if the user has been granted to read system objects. USER_PROCEDURES shows only the objects owned by the current user, and then can read from the current container only.

For the ease of the definition, in 12c all the joins on the underlying tables(such as procedureinfo$, user$, obj$) is done by an intermediate view such as INT$DBA_PROCEDURES which is defined as EXTENDED DATA link to read from CDB$ROOT in addition to the local table. Then DBA_PROCEDURES, ALL_PROCEDURES and USER_PROCEDURES are defined on top of it with the required where clause to filter out owner and privilege accessibility.

INT$INT$ Extended Data views

In this post, I’ll detail the special case of DBA_CONSTRAINTS because things are more complex to get the multitenant architecture behaving the same as the non-CDB.

There are several types of constraints which are identified with the CONSTRAINT_TYPE column of DBA_CONSTRAINTS, or the TYPE# of the underlying table CDEF#

Here, I query the underlying table with the CONTAINER() function to see what is stored in each container:

SQL> select decode(type#,1,'C',2,'P',3,'U',4,'R',5,'V',6,'O',7,'C',8,'H',9,'F',10,'F',11,'F',13,'F','?') constraint_type,
  2  type#,con_id,count(*) from containers(cdef$)
  3  group by type#,con_id order by type#,con_id;
 
CONSTRAINT_TYPE     TYPE#   CON_ID   COUNT(*)
---------------     -----   ------   --------
C                       1        1         74
C                       1        3         74
P                       2        1        843
P                       2        3        844
U                       3        1        238
U                       3        3        238
R                       4        1        324
R                       4        3        324
V                       5        1         11
O                       6        1        172
O                       6        3         26
C                       7        1       5337
C                       7        3       5337
F                      11        1         11
F                      11        3         11
?                      12        1          3
?                      12        3          3

I have very few user objects in this database. CON_ID=1 is CDB$ROOT and CON_ID=3 is my PDB. What we can see here is that we have nearly the same number of rows in both containers for the following constraint types: C (check constraint on a table), P (primary key), U (unique key), R (referential integrity), and other types related to tables. And some types have most of their rows in CDB$ROOT only: V (check option on views), R (read only on views)

That’s an implementation specificity of the multitenant architecture which makes things more complex for the dictionary views. For some objects (such as procedures and views) the metadata is stored in only one container: system objects have all their information in CDB$ROOT and the PDB has only a link which is a dummy row in OBJ$ which mentions the sharing (such as metadata link), owner and name (to match to the object in CDB$ROOT), and a signature (to verify that the DDL creating the object is the same). But other objects (such as tables) have their information duplicated in all containers for system objects (CDB$ROOT, PDB$SEED and all user PDBs). This is the reason why we see rows in both containers for constraint definition when they are related to a table.

Example on view constraint

I’ll take a constraint on system view as an example: constraint SYS_C003357 on table SYS.DBA_XS_SESSIONS

SQL> select owner,object_name,object_type,sharing from dba_objects where owner='SYS' and object_name='DBA_XS_SESSIONS';
 
OWNER   OBJECT_NAME       OBJECT_TYPE   SHARING
-----   -----------       -----------   -------
SYS     DBA_XS_SESSIONS   VIEW          METADATA LINK
 
SQL> select owner,table_name,constraint_type,constraint_name from containers(dba_constraints) where owner='SYS' and table_name='DBA_XS_SESSIONS'  and rownum=1;
 
OWNER   TABLE_NAME        CONSTRAINT_TYPE   CONSTRAINT_NAME
-----   ----------        ---------------   ---------------
SYS     DBA_XS_SESSIONS   O                 SYS_C003357

I’m looking at the dependencies for the DBA_CONSTRAINTS view:

SQL> select owner,name,referenced_owner,referenced_name from dba_dependencies where owner='SYS' and name='DBA_CONSTRAINTS' and type='VIEW';
 
OWNER   NAME              REFERENCED_OWNER   REFERENCED_NAME
-----   ----              ----------------   ---------------
SYS     DBA_CONSTRAINTS   SYS                GETLONG
SYS     DBA_CONSTRAINTS   SYS                INT$DBA_CONSTRAINTS

So the DBA_CONSTRAINT is a view on INT$DBA_CONSTRAINTS as we have seen above. However, this view is not directly reading the tables but another view:

SQL> select owner,name,referenced_owner,referenced_name from dba_dependencies where owner='SYS' and name='INT$DBA_CONSTRAINTS' and type='VIEW';
 
OWNER   NAME                  REFERENCED_OWNER   REFERENCED_NAME
-----   ----                  ----------------   ---------------
SYS     INT$DBA_CONSTRAINTS   SYS                GETLONG
SYS     INT$DBA_CONSTRAINTS   SYS                INT$INT$DBA_CONSTRAINTS

Here is our additional INT$INT$ view which is reading the tables:

SQL> select owner,name,referenced_owner,referenced_name from dba_dependencies where owner='SYS' and name='INT$INT$DBA_CONSTRAINTS' and type='VIEW';
 
OWNER   NAME                      REFERENCED_OWNER   REFERENCED_NAME
-----   ----                      ----------------   ---------------
SYS     INT$INT$DBA_CONSTRAINTS   SYS                USER$
SYS     INT$INT$DBA_CONSTRAINTS   SYS                CDEF$
SYS     INT$INT$DBA_CONSTRAINTS   SYS                OBJ$
SYS     INT$INT$DBA_CONSTRAINTS   SYS                CON$
SYS     INT$INT$DBA_CONSTRAINTS   SYS                _CURRENT_EDITION_OBJ
SYS     INT$INT$DBA_CONSTRAINTS   SYS                _BASE_USER
SYS     INT$INT$DBA_CONSTRAINTS   SYS                GETLONG

In summary, the EXTENDED DATA view which reads the tables on each container (CDB$ROOT and PDB) is here the INT$INT$DBA_CONSTRAINTS and the INT$DBA_CONSTRAINTS is another intermediate one before the DBA_CONSTRAINTS view.

SQL> select owner,object_name,object_type,sharing from dba_objects where object_name in ('DBA_CONSTRAINTS','INT$DBA_CONSTRAINTS','INT$INT$DBA_CONSTRAINTS') order by object_id desc;
 
OWNER    OBJECT_NAME               OBJECT_TYPE   SHARING
-----    -----------               -----------   -------
PUBLIC   DBA_CONSTRAINTS           SYNONYM       METADATA LINK
SYS      DBA_CONSTRAINTS           VIEW          METADATA LINK
SYS      INT$DBA_CONSTRAINTS       VIEW          METADATA LINK
SYS      INT$INT$DBA_CONSTRAINTS   VIEW          EXTENDED DATA LINK

In this example, we don’t understand the reason for the additional intermediate view because the return all the same number of rows in each container:

SQL> select con_id,constraint_type,constraint_name from containers(INT$INT$DBA_CONSTRAINTS) where OWNER='SYS' and constraint_name='SYS_C003357'
 
  CON_ID CONSTRAINT_TYPE   CONSTRAINT_NAME
  ------ ---------------   ---------------
       1 O                 SYS_C003357
       3 O                 SYS_C003357
 
SQL> select con_id,constraint_type,constraint_name from containers(INT$DBA_CONSTRAINTS) where OWNER='SYS' and constraint_name='SYS_C003357'
 
  CON_ID CONSTRAINT_TYPE   CONSTRAINT_NAME
  ------ ---------------   ---------------
       1 O                 SYS_C003357
       3 O                 SYS_C003357
 
SQL> select con_id,constraint_type,constraint_name from containers(DBA_CONSTRAINTS) where OWNER='SYS' and constraint_name='SYS_C003357'
 
  CON_ID CONSTRAINT_TYPE   CONSTRAINT_NAME
  ------ ---------------   ---------------
       1 O                 SYS_C003357
       3 O                 SYS_C003357

The difference is only a few additional columns from the object definition (OWNERID,OBJECT_ID,OBJECT_TYPE#,SHARING) in the INT$ and INT$INT$ which are not selected in the final view:

SQL> select * from containers(INT$INT$DBA_CONSTRAINTS) where OWNER='SYS' and constraint_name='SYS_C003357'
 
OWNER     OWNERID CONSTRAINT_NAME   CONSTRAINT_TYPE   TABLE_NAME          OBJECT_ID   OBJECT_TYPE# SEARCH_CONDITION_VC   R_OWNER   R_CONSTRAINT_NAME   DELETE_RULE   STATUS    DEFERRABLE       DEFERRED    VALIDATED       GENERATED        BAD   RELY   LAST_CHANGE   INDEX_OWNER   INDEX_NAME   INVALID   VIEW_RELATED     SHARING   ORIGIN_CON_ID   CON_ID
-----     ------- ---------------   ---------------   ----------          ---------   ------------ -------------------   -------   -----------------   -----------   ------    ----------       --------    ---------       ---------        ---   ----   -----------   -----------   ----------   -------   ------------     -------   -------------   ------
SYS             0 SYS_C003357       O                 DBA_XS_SESSIONS         10316              4                                                                   ENABLED   NOT DEFERRABLE   IMMEDIATE   NOT VALIDATED   GENERATED NAME                26-JAN-17                                                                 1               1        1
SYS             0 SYS_C003357       O                 DBA_XS_SESSIONS         10316              4                                                                   ENABLED   NOT DEFERRABLE   IMMEDIATE   NOT VALIDATED   GENERATED NAME                26-JAN-17                                                                 1               1        3
 
SQL> select * from containers(INT$DBA_CONSTRAINTS) where OWNER='SYS' and constraint_name='SYS_C003357'
 
OWNER     OWNERID CONSTRAINT_NAME   CONSTRAINT_TYPE   TABLE_NAME          OBJECT_ID   OBJECT_TYPE# SEARCH_CONDITION_VC   R_OWNER   R_CONSTRAINT_NAME   DELETE_RULE   STATUS    DEFERRABLE       DEFERRED    VALIDATED       GENERATED        BAD   RELY   LAST_CHANGE   INDEX_OWNER   INDEX_NAME   INVALID   VIEW_RELATED     SHARING   ORIGIN_CON_ID   CON_ID
-----     ------- ---------------   ---------------   ----------          ---------   ------------ -------------------   -------   -----------------   -----------   ------    ----------       --------    ---------       ---------        ---   ----   -----------   -----------   ----------   -------   ------------     -------   -------------   ------
SYS             0 SYS_C003357       O                 DBA_XS_SESSIONS         10316              4                                                                   ENABLED   NOT DEFERRABLE   IMMEDIATE   NOT VALIDATED   GENERATED NAME                26-JAN-17                                                                 1               1        1
SYS             0 SYS_C003357       O                 DBA_XS_SESSIONS         10316              4                                                                   ENABLED   NOT DEFERRABLE   IMMEDIATE   NOT VALIDATED   GENERATED NAME                26-JAN-17                                                                 1               1        3
 
SQL> select * from containers(DBA_CONSTRAINTS) where OWNER='SYS' and constraint_name='SYS_C003357'
 
OWNER   CONSTRAINT_NAME   CONSTRAINT_TYPE   TABLE_NAME        SEARCH_CONDITION_VC   R_OWNER   R_CONSTRAINT_NAME   DELETE_RULE   STATUS    DEFERRABLE       DEFERRED    VALIDATED       GENERATED        BAD   RELY   LAST_CHANGE   INDEX_OWNER   INDEX_NAME   INVALID   VIEW_RELATED     ORIGIN_CON_ID   CON_ID
-----   ---------------   ---------------   ----------        -------------------   -------   -----------------   -----------   ------    ----------       --------    ---------       ---------        ---   ----   -----------   -----------   ----------   -------   ------------     -------------   ------
SYS     SYS_C003357       O                 DBA_XS_SESSIONS                                                                     ENABLED   NOT DEFERRABLE   IMMEDIATE   NOT VALIDATED   GENERATED NAME                26-JAN-17                                                                       1        1
SYS     SYS_C003357       O                 DBA_XS_SESSIONS                                                                     ENABLED   NOT DEFERRABLE   IMMEDIATE   NOT VALIDATED   GENERATED NAME                26-JAN-17   

If we look at the INT$DBA_CONSTRAINTS definition we see some filters on those object definition:

SQL> ddl INT$DBA_CONSTRAINTS
 
  CREATE OR REPLACE FORCE NONEDITIONABLE VIEW "SYS"."INT$DBA_CONSTRAINTS" ("OWNER", "OWNERID", "CONSTRAINT_NAME", "CONSTRAINT_TYPE", "TABLE_NAME", "OBJECT_ID", "OBJECT_TYPE#", "SEARCH_CONDITION", "SEARCH_CONDITION_VC", "R_OWNER", "R_CONSTRAINT_NAME", "DELETE_RULE", "STATUS", "DEFERRABLE", "DEFERRED", "VALIDATED", "GENERATED", "BAD", "RELY", "LAST_CHANGE", "INDEX_OWNER", "INDEX_NAME", "INVALID", "VIEW_RELATED", "SHARING", "ORIGIN_CON_ID") AS
  select OWNER, OWNERID, CONSTRAINT_NAME, CONSTRAINT_TYPE,
       TABLE_NAME, OBJECT_ID, OBJECT_TYPE#, SEARCH_CONDITION,
       SEARCH_CONDITION_VC, R_OWNER, R_CONSTRAINT_NAME, DELETE_RULE, STATUS,
       DEFERRABLE, DEFERRED, VALIDATED, GENERATED,
       BAD, RELY, LAST_CHANGE, INDEX_OWNER, INDEX_NAME,
       INVALID, VIEW_RELATED, SHARING, ORIGIN_CON_ID
from   INT$INT$DBA_CONSTRAINTS INT$INT$DBA_CONSTRAINTS
where  INT$INT$DBA_CONSTRAINTS.OBJECT_TYPE# = 4         /* views */
       OR (INT$INT$DBA_CONSTRAINTS.OBJECT_TYPE# = 2     /* tables */
           AND (INT$INT$DBA_CONSTRAINTS.ORIGIN_CON_ID
                = TO_NUMBER(SYS_CONTEXT('USERENV', 'CON_ID'))));

For views (OBJECT_TYPE#=4) there is no filter, which explains why we see the same number of rows in the previous example. But for tables (OBJECT_TYPE#=2) there’s an additional filter to keep the row from the current container only.

Example on table constraint

Then, I’ll take another example with a constraint definition for a table:

SQL> select owner,object_name,object_type,sharing from dba_objects where owner='SYS' and object_name='RXS$SESSIONS';
 
OWNER   OBJECT_NAME    OBJECT_TYPE   SHARING
-----   -----------    -----------   -------
SYS     RXS$SESSIONS   TABLE         METADATA LINK
 
SQL> select owner,table_name,constraint_type,constraint_name from dba_constraints where owner='SYS' and table_name='RXS$SESSIONS' and rownum=1;
 
OWNER   TABLE_NAME     CONSTRAINT_TYPE   CONSTRAINT_NAME
-----   ----------     ---------------   ---------------
SYS     RXS$SESSIONS   C                 SYS_C003339

From the INT$INT$ view, we have a duplicate when we query on a PDB because for tables the PDB not only holds a dummy row in OBJ$ but full information about the table is duplicated in other tables such as TAB$ and CDEF$:

SQL> select con_id,constraint_type,constraint_name from containers(INT$INT$DBA_CONSTRAINTS) where OWNER='SYS' and constraint_name='SYS_C003339'
 
  CON_ID CONSTRAINT_TYPE   CONSTRAINT_NAME
  ------ ---------------   ---------------
       1 C                 SYS_C003339
       3 C                 SYS_C003339
       3 C                 SYS_C003339

This is the reason for the additional intermediate view: filtering out those duplicate by removing the rows from CDB$ROOT when queried from a PDB.

SQL> select con_id,constraint_type,constraint_name from containers(INT$DBA_CONSTRAINTS) where OWNER='SYS' and constraint_name='SYS_C003339'
 
  CON_ID CONSTRAINT_TYPE   CONSTRAINT_NAME
  ------ ---------------   ---------------
       1 C                 SYS_C003339
       3 C                 SYS_C003339

Thanks to that, the duplicates are not visible to the end-user views DBA_CONSTRAINTS and PDB_CONSTRAINTS.

You may wonder why only DBA_CONSTRAINTS needs this views and not DBA_TABLES, DBA_INDEXES or DBA_TAB_COLUMNS? That’s because all information about system tables and indexes are replicated in all PDBs and then there is no need for EXTENDED DATA and context switches. DBA_CONSTRAINT has the particularity of showing information about tables and views, which implement the metadata links in a different way.

L’article Multitenant internals: INT$ and INT$INT$ views est apparu en premier sur dbi Blog.

.
The documentation says that for Reduction of duplication and Ease of database upgrade the Oracle-supplied objects such as data dictionary table definitions and PL/SQL packages are represented only in the root.

Unfortunately, this is only partly true. System PL/SQL packages are only in root but system table definition are replicated into all PDBs.

This post is an extension of a previous blog post which was on 12cR1. This one is on 12cR2.

As I did at Open World and will do at DOAG, I show multitenant internals by creating a metadata link procedure. When I do a simple ‘describe’ when connected to a PDB, the sql_trace shows that the session switches to the CDB$ROOT to get the procedure information:

*** 2017-11-05T16:17:36.339126+01:00 (CDB$ROOT(1))
=====================
PARSING IN CURSOR #140420856738440 len=143 dep=1 uid=0 oct=3 lid=0 tim=101728244788 hv=2206365737 ad='7f60a7f0' sqlid='9fjf75a1s4y19'
select procedure#,procedurename,properties,itypeobj#, properties2 from procedureinfo$ where obj#=:1 order by procedurename desc, overload# desc
END OF STMT

All information about the system PL/SQL procedures is stored in the root only. The PDB has only a dummy row in OBJ$ to mention that it is a metadata link. And this is why you pay for the multitenant option: consolidation of all system dictionary objects into the root only. You save space (on disk and related memory) and you have only one place to upgrade.

But this is implemented only for some objects, like PL/SQL procedures, but not for others like table and indexes. If you ‘describe’ a metadata link table when connected to a PDB you will not see any switch to CDB$ROOT in the sql_trace:

*** 2017-11-05T13:01:53.541231+01:00 (PDB1(3))
PARSING IN CURSOR #139835947128936 len=86 dep=1 uid=0 oct=3 lid=0 tim=98244321664 hv=2195287067 ad='75f823b8' sqlid='32bhha21dkv0v'
select col#,intcol#,charsetid,charsetform from col$ where obj#=:1 order by intcol# asc
END OF STMT
PARSE #139835947128936:c=0,e=158,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=3765558045,tim=98244321664
BINDS #139835947128936:
 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7f2e124fef10  bln=22  avl=03  flg=05
  value=747
EXEC #139835947128936:c=1000,e=603,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=3765558045,tim=98244322311
FETCH #139835947128936:c=0,e=15,p=0,cr=3,cu=0,mis=0,r=1,dep=1,og=4,plh=3765558045,tim=98244322342
FETCH #139835947128936:c=0,e=1,p=0,cr=0,cu=0,mis=0,r=1,dep=1,og=4,plh=3765558045,tim=98244322356
FETCH #139835947128936:c=0,e=4,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=3765558045,tim=98244322369
STAT #139835947128936 id=1 cnt=2 pid=0 pos=1 obj=0 op='SORT ORDER BY (cr=3 pr=0 pw=0 str=1 time=16 us cost=3 size=234 card=13)'
STAT #139835947128936 id=2 cnt=2 pid=1 pos=1 obj=21 op='TABLE ACCESS CLUSTER COL$ (cr=3 pr=0 pw=0 str=1 time=11 us cost=2 size=234 card=13)'
STAT #139835947128936 id=3 cnt=1 pid=2 pos=1 obj=3 op='INDEX UNIQUE SCAN I_OBJ# (cr=2 pr=0 pw=0 str=1 time=6 us cost=1 size=0 card=1)'
CLOSE #139835947128936:c=0,e=1,dep=1,type=3,tim=98244322439

Here all information about the columns is read from COL$ in the PDB. And if you look at TAB$ (tables), COL$ (table columns), IND$ (indexes), CONS$ and CDEF$ (constraints), you will see that they contain rows in a PDB where no user objects have been created. This is the case for all information related to tables: they are stored in CDB$ROOT and replicated into all other containers: PDB$SEED and all user created PDB. Only the information related to non-data objects, are stored only in one container.

I’ve run a query to count the rows in CDB$ROOT and PDB$SEED and here is the result:

All rows in OBJ$ are replicated, which is expected because this is where the metadata link information is stored. But you see also all information related to tables that are also replicated, such as the 100000+ columns in COL$. And this is the reason why you do not see a big consolidation benefit when you look at the size of the SYSTEM tablespace in pluggable databases which do no contain any user data:

List of Permanent Datafiles
===========================
File Size(MB) Tablespace           RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1    820      SYSTEM               YES     /u01/oradata/CDB1A/system01.dbf
3    630      SYSAUX               NO      /u01/oradata/CDB1A/sysaux01.dbf
4    80       UNDOTBS1             YES     /u01/oradata/CDB1A/undotbs01.dbf
5    250      PDB$SEED:SYSTEM      NO      /u01/oradata/CDB1A/pdbseed/system01.dbf
6    390      PDB$SEED:SYSAUX      NO      /u01/oradata/CDB1A/pdbseed/sysaux01.dbf
7    5        USERS                NO      /u01/oradata/CDB1A/users01.dbf
8    100      PDB$SEED:UNDOTBS1    NO      /u01/oradata/CDB1A/pdbseed/undotbs01.dbf
9    270      PDB1:SYSTEM          YES     /u01/oradata/CDB1A/PDB1/system01.dbf
10   440      PDB1:SYSAUX          NO      /u01/oradata/CDB1A/PDB1/sysaux01.dbf
11   100      PDB1:UNDOTBS1        YES     /u01/oradata/CDB1A/PDB1/undotbs01.dbf
12   5        PDB1:USERS           NO      /u01/oradata/CDB1A/PDB1/users01.dbf

Here I have 250MB in PDB$SEED which is supposed to contain only links to the 820GB SYSTEM tablespace, but there is a lot more than that.

So, basically, not all the dictionary is consolidated in multitenant but only the non-data part such as those PL/SQL packages and the dictionary views definition. You can think about the multitenant option consolidation as an extension to sharing the Oracle Home among several databases. It concerns the software part only. But the part of the dictionary which contains data about system objects is replicated into all containers, and is read locally without a context switch. This also means that a patch or upgrade on them has to be run in all containers.

With the fact that some information is replicated and some are not, comes the complexity to manage that in the dictionary views, and this will be the subject of the next blog post about INT$INT$ views.

L’article Multitenant dictionary: what is consolidated and what is not est apparu en premier sur dbi Blog.