{"id":13815,"date":"2020-03-26T22:56:41","date_gmt":"2020-03-26T21:56:41","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/oracle-disables-your-multitenant-option-when-you-run-on-ec2\/"},"modified":"2020-03-26T22:56:41","modified_gmt":"2020-03-26T21:56:41","slug":"oracle-disables-your-multitenant-option-when-you-run-on-ec2","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/oracle-disables-your-multitenant-option-when-you-run-on-ec2\/","title":{"rendered":"Oracle disables your multitenant option when you run on EC2"},"content":{"rendered":"

I have installed Oracle 19.6 on an EC2 for our Multitenant Workshop<\/a> training. And of course, during the workshop we create a lot of PDBs. If you don’t have paid for the Enterprise Edition plus the Multitenant Option you can create at most 3 pluggable database. But with this option you can create up to 252 pluggable databases. Does it worth the price, which according to the public price list<\/a> is USD 47,500 + 17,500 per processor, which means per-core because Oracle doesn’t count the core factor when your Intel processors are in AWS Cloud (according to the Authorized Cloud Environments<\/a> paper)? Probably not because Oracle detects where you run and bridles some features depending whether you are on the Dark or the Light Side of the public cloud (according to their criteria of course).
\n
\nAt one point I have 3 pluggable databases in my CDB:<\/p>\n

\nSQL> show pdbs\n   CON_ID     CON_NAME    OPEN MODE    RESTRICTED\n_________ ____________ ____________ _____________\n        2 PDB$SEED     READ ONLY    NO\n        3 CDB1PDB01    MOUNTED\n        4 CDB1PDB03    MOUNTED\n        5 CDB1PDB02    MOUNTED\n<\/code><\/pre>\n
I want to create a 4th one:<\/p>\n
\nSQL> create pluggable database CDB1PDB04 from CDB1PDB03;\n\ncreate pluggable database CDB1PDB04 from CDB1PDB03\n                          *\nERROR at line 1:\nORA-65010: maximum number of pluggable databases created\n<\/code><\/pre>\n
It fails. The maximum number of pluggable databases is defined by MAX_PDBS, but I defined nothing in my SPFILE:<\/p>\n
\nSQL> show spparameter max_pdbs\nSID NAME     TYPE    VALUE\n--- -------- ------- -----\n*   max_pdbs integer\n<\/code><\/pre>\n
I thought that the default was 4098 (which is incorrect anyway as you cannot create more than 4096) but it is actually 5 here:<\/p>\n
\nSQL> show parameter max_pdbs\nNAME     TYPE    VALUE\n-------- ------- -----\nmax_pdbs integer 5\n<\/code><\/pre>\n
Ok… this parameter is supposed to count the number of user pluggable databases (the ones with CON_ID>2) and I have 3 of them here. The limit is 5 and I have an error mentioning that I’ve reached the limit. That’s not the first time I see wrong maths<\/a> with this parameter. But there’s worse as I cannot change it:<\/p>\n
\nSQL> alter system set max_pdbs=6;\n\nalter system set max_pdbs=6\n *\nERROR at line 1:\nORA-02097: parameter cannot be modified because specified value is invalid\nORA-65334: invalid number of PDBs specified\n<\/code><\/pre>\n
I can change it in the SPFILE but it doesn’t help me to create more pluggable databases:<\/p>\n
\nSQL> alter system set max_pdbs=200 scope=spfile;\n\nSystem altered.\n\nSQL> startup force;\n\nTotal System Global Area   2147482744 bytes\nFixed Size                    9137272 bytes\nVariable Size               587202560 bytes\nDatabase Buffers           1543503872 bytes\nRedo Buffers                  7639040 bytes\nDatabase mounted.\nDatabase opened.\n\nSQL> show parameter max_pdbs\nNAME     TYPE    VALUE\n-------- ------- -----\nmax_pdbs integer 200\n\nSQL> create pluggable database CDB1PDB04 from CDB1PDB03;\n\ncreate pluggable database CDB1PDB04 from CDB1PDB03\n                          *\nERROR at line 1:\nORA-65010: maximum number of pluggable databases created\n<\/code><\/pre>\n
Something bridles me. There’s a MOS Note ORA-65010 When Oracle Database Hosted on AWS Cloud (Doc ID 2328600.1)<\/a> about the same problem but that’s in 12.1.0.2 (before MAX_PDBS was introduced) which is supposed to be fixed in AUG 2017 PSU. But here I am 3 years later in 19.6 (the January 2020 Release Update for the latest version available on-premises).<\/p>\n
So, Oracle limits the number of pluggable databases when we are on a public cloud provider which is not the Oracle Public Cloud. This limitation is not documented in the licensing documentation<\/a> which mentions 252 as the Enterprise Edition limit, and I see nothing about “Authorized Cloud Environments” limitations for this item. This, and the fact that it can come and go with Release Updates put customers at risk when running on AWS EC2: financial risk and availability risk. I think there are only two choices, on long term, when you want to run your database on a cloud: go to Oracle Cloud or leave for another Database.<\/p>\n
How does the Oracle instance know on which public cloud you run? All cloud platforms provide some metadata through HTTP api. I have straced all sendto() and recvfrom() system calls when starting the instance:<\/p>\n
\nstrace -k -e trace=recvfrom,sendto -yy -s 1000 -f -o trace.trc sqlplus \/ as sysdba <<<'startup force'\n<\/code><\/pre>\n
And I searched for Amazon and AWS here:<\/p>\n
\"\"<\/a><\/p>\n
This is clear: the instance has a function to detect the cloud provider (kgcs_clouddb_provider_detect) when initializing the SGA in a multitenant architecture (kpdbInitSga) with the purpose of detecting non-oracle clouds (kscs_is_non_oracle_cloud). This queries the AWS metadata (documented on Retrieving Instance Metadata<\/a>):<\/p>\n
\n[oracle@ora-cdb-1 ~]$ curl http:\/\/169.254.169.254\/latest\/meta-data\/services\/domain\namazonaws.com\/\n<\/code><\/pre>\n
When Oracle software sees the name of the enemy in the domain name amazonaws.com, it sets an internal limit for the number of pluggable databases that overrides the MAX_PDBS setting. Ok, I don’t need this metadata and I’m root on EC2 so my simple workaround is to block this metadata API:<\/p>\n
\n[root@ora-cdb-1 ~]# iptables -A OUTPUT -d 169.254.169.254  -j REJECT\n[root@ora-cdb-1 ~]# iptables -L\nChain OUTPUT (policy ACCEPT)\ntarget     prot opt source               destination\nREJECT     udp  --  anywhere             10.0.0.2             udp dpt:domain reject-with icmp-port-unreachable\nREJECT     all  --  anywhere             10.0.0.2             reject-with icmp-port-unreachable\n<\/code><\/pre>\n
Then restart the instance and it works: I can set or reset MAX_PDBS and create more pluggable databases.<\/p>\n
I can remove the rule<\/p>\n
\n[root@ora-cdb-1 ~]# iptables -D OUTPUT -d 169.254.169.254  -j REJECT\n<\/code><\/pre>\n
If, for watever reason I want to revert back.<\/p>\n
Finally, because they had many bugs<\/a> with the MAX_PDBS soft limit, there’s a parameter to disable it and this disables also the hard limit:<\/p>\n
\nSQL> alter system set \"_cdb_disable_pdb_limit\"=true scope=spfile;\nSystem altered.\n<\/code><\/pre>\n
Thanks to Mauricio Melnik for the heads-up on that:<\/p>\n
\n
MOS 2538172.1  _cdb_disable_pdb_limit=true …. maybe?<\/p>\n
— MM (@maurimelnik) March 26, 2020<\/a><\/p><\/blockquote>\n