Archives des Oracle database appliance - dbi Blog https://www.dbi-services.com/blog/tag/oracle-database-appliance/ Wed, 25 Jan 2023 09:17:44 +0000 en-US hourly 1 https://www.dbi-services.com/blog/wp-content/uploads/sites/2/2025/05/cropped-favicon_512x512px-min-32x32.png Archives des Oracle database appliance - dbi Blog https://www.dbi-services.com/blog/tag/oracle-database-appliance/ 32 32 Deploying 11.2.0.4 on a recent ODA https://www.dbi-services.com/blog/deploying-11-2-0-4-on-a-recent-oda/ https://www.dbi-services.com/blog/deploying-11-2-0-4-on-a-recent-oda/#comments Wed, 25 Jan 2023 09:16:23 +0000 https://www.dbi-services.com/blog/?p=21984 How to deploy 11gR2 database on a modern ODA

L’article Deploying 11.2.0.4 on a recent ODA est apparu en premier sur dbi Blog.

]]> Introduction

11gR2 should be retired now, and there is normally no need to deploy a new database using this version anymore. But in the real world, you may need to do that for some reasons. 11gR2 is no more supported on Oracle Database Appliance, but this is the way you can setup a “brand new” Oracle Database 11gR2 environment on ODA. Obviously, I would only recommend doing this if there is no other options.

History

The first ODAs were designed for 11gR2, but support for this version disappeared nearly 2 years ago. There are still ODAs running 11gR2 databases, but it’s mostly old ODAs deployed several years ago. Applying the latest ODA patch on top of 11gR2 databases is not a problem, your 11gR2 databases won’t be patched although every other components will. Therefore, running the most recent 19.17 patch with 11gR2 databases is OK. But deploying a new infrastructure with X9-2 ODAs running 11gR2 databases wouldn’t make sense and is not supported. Furthermore, ODA development’s team now focus on modern features, mainly tied with Oracle 19c. Don’t expect to run these new features on old 11gR2 databases. Most of the time it won’t work correctly as Oracle doesn’t bother anymore testing on 11gR2.

2 solutions for running 11gR2 databases on ODA

If you absolutely need 11gR2, you currently have 2 solutions.

The first one is deploying an old set of binaries from an old patch, for example 19.10. It’s the easiest way to put 11gR2 on your ODA:

cd /opt/dbi/
unzip p23494997_1910000_Linux-x86-64.zip
odacli update-repository -f /opt/dbi/odacli-dcs-19.10.0.0.0-210115-DB-11.2.0.4.zip 
odacli create-dbhome -v 11.2.0.4.210119
odacli list-dbhomes 

ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
2a00d3bb-b042-4720-94a2-bef13bfaf5f5     OraDB19000_home1     19.15.0.0.220419                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 CONFIGURED
4f56e25e-e228-4cc3-b827-b15b66c67143     OraDB19000_home4     19.16.0.0.220719                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4 CONFIGURED
306d949e-6a61-4d7f-83e2-b023a9c47586     OraDB11204_home2     11.2.0.4.210119                          /u01/app/odaorahome/oracle/product/11.2.0.4/dbhome_2 CREATING

The second one we will focus on is creating a new user-managed VM and manually deploy an 11gR2 DB home. I would definitely prefer this solution because I don’t want to use unsupported version on my ODA: I want to keep everything clean and supported. Creating a user-managed VM needs some work, but this VM won’t have any link to my ODA system, everything will be running on its own inside the VM.

License considerations

You should be careful when using user-managed VMs with Oracle databases inside. If you run on Standard Edition, you normally have 1 license per socket on your system, and you can use as may cores as you want. When using Enterprise Edition, you will limit the available number of cores on your ODA with odacli update-cpucore, or you will use CPU Pools for DB Systems and run all your DB Systems in these CPU Pools. Configuring a VM CPU pool (which is different from DB Systems’ CPU pool) for databases in user-managed VMs is not compliant with the Oracle licensing model. You’d better limiting the total CPU-cores of your ODA in this case. For example, if you need 6 cores for 19c DB Systems and 2 cores for your 11gR2, configure 8 cores on your ODA with odacli update-cpucore and then configure the 2 CPU pools accordingly.

Test environment

My test environment is based on a brand new ODA X9-2S deployed with 19.16 and using Standard Edition. I will use 19c DB Systems for most databases, and I will need a user-managed VM for 11gR2. The purpose of this user-managed VM is to decomission an old 11gR2 server and put all Oracle databases into the ODA. The application linked to this 11gR2 database will also be decomissioned, but later. It’s why there is no plan to migrate this database to a newer version.

Setting up the user-managed 11gR2 VM

I will need a VM CPU pool for this “old” system:

odacli create-cpupool -n cpupool4olddbs -c 2 -vm
odacli list-cpupools
Name                  Type                Configured on Cores Associated resources     Created                   Updated
--------------------  ------------------  ------------- ----- -----------------------  ------------------------  ----------
cpupool4olddbs        VM                  srv-bd3        2    NONE        2022-11-29 14:50:28 CET   2022-11-29 14:50:28 CET
cpupool4dbsystems     DB_SYSTEM_SHARED    srv-bd3        8    NONE        2022-11-29 14:49:55 CET   2022-11-29 14:49:55 CET

My VM will need filesystems, I first need to create 2 VM storages, one for DATA and one for Recovery Area:

odacli create-vmstorage -n VMsDATA -dg DATA -s 300G
odacli create-vmstorage -n VMsRECO -dg RECO -s 100G

odacli list-vmstorages
Name                  Disk group       Volume name      Volume device                   Size        Mount Point                          Created                   Updated
--------------------  ---------------  ---------------  ------------------------------  ----------  -----------------------------------  ------------------------  ------------------------
VMsRECO               RECO             VMSRECO          /dev/asm/vmsreco-115            100.00 GB   /u05/app/sharedrepo/vmsreco          2022-11-30 09:34:38 CET   2022-11-30 09:34:38 CET
VMsDATA               DATA             VMSDATA          /dev/asm/vmsdata-211            300.00 GB   /u05/app/sharedrepo/vmsdata          2022-11-30 09:34:04 CET   2022-11-30 09:34:04 CET

Now I will create vdisks in these VM storages:

odacli create-vdisk -n prlinux11-data -vms VMsDATA -s 200G
odacli create-vdisk -n prlinux11-reco -vms VMsRECO -s 70G

odacli list-vdisks

Name                  VM storage            Size        Shared      Sparse      Created                   Updated
--------------------  --------------------  ----------  ----------  ----------  ------------------------  ------------------------
prlinux11-reco         VMsRECO               70.00 GB    NO          NO          2022-11-30 09:39:51 CET   2022-11-30 09:39:51 CET
prlinux11-data         VMsDATA               200.00 GB   NO          NO          2022-11-30 09:41:06 CET   2022-11-30 09:41:06 CET

Now, let’s create the VM. I will use an Oracle Linux distribution provided as an ISO file. Creating this VM will create a boot disk on the VM storage VMsDATA, and connect the 2 vdisks previously created. I will use my ODA’s IP address to map the VNC port.

odacli create-vm -n PRLINUX11 -m 32G -src /mnt/nas/V1009690-01.iso -vd prlinux11-data,prlinux11-reco -vc 2 -cp cpupool4olddbs -vn prnetwork -vms VMsDATA -s 50G -g "vnc,listen=10.146.107.31"

odacli describe-job -i 5d94d0b1-da6c-43d0-89ca-98f6e5a89cfa

Job details
----------------------------------------------------------------
                     ID:  5d94d0b1-da6c-43d0-89ca-98f6e5a89cfa
            Description:  VM PRLINUX11 creation
                 Status:  Success
                Created:  November 30, 2022 10:05:41 AM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validate dependency resources            November 30, 2022 10:05:41 AM CET   November 30, 2022 10:05:41 AM CET   Success
Validate resource allocations            November 30, 2022 10:05:41 AM CET   November 30, 2022 10:05:41 AM CET   Success
Allocate resources                       November 30, 2022 10:05:41 AM CET   November 30, 2022 10:05:41 AM CET   Success
Provision new VM                         November 30, 2022 10:05:41 AM CET   November 30, 2022 10:05:43 AM CET   Success
Add VM to Clusterware                    November 30, 2022 10:05:43 AM CET   November 30, 2022 10:05:44 AM CET   Success
Save domain in ACFS                      November 30, 2022 10:05:44 AM CET   November 30, 2022 10:05:44 AM CET   Success
Create VM metadata                       November 30, 2022 10:05:44 AM CET   November 30, 2022 10:05:44 AM CET   Success
Persist metadata                         November 30, 2022 10:05:44 AM CET   November 30, 2022 10:05:44 AM CET   Success

odacli describe-vm -n PRLINUX11 | grep Display
             Display Port:  10.146.107.31:3

Creating a VM is fast as nothing is really created.

Now I can use a VNC client connected to 10.146.107.31:5903 and deploy a Linux distribution. I would recommend using the same OS as your ODA, meaning an Oracle Linux 7.9, but you can use an older one if needed. Linux setup on this VM is quite typical, additional settings and packages will be deployed using preinstall package provided by Oracle.

Once the Linux is deployed, let’s check the disks and configure LVM according to OFA naming:

fdisk -l /dev/vdb | grep GB
Disk /dev/vdb: 214.7 GB, 214748364800 bytes, 419430400 sectors
fdisk -l /dev/vdc | grep GB
Disk /dev/vdc: 75.2 GB, 75161927680 bytes, 146800640 sectors

pvcreate /dev/vdb
pvcreate /dev/vdc

vgcreate vg_oradata /dev/vdb
vgcreate vg_orareco /dev/vdc

lvcreate -L 30G -n lv_u01 vg_oradata
lvcreate -L 160G -n lv_data vg_oradata
lvcreate -L 60G -n lv_reco vg_orareco

mkdir /u01
mkdir -p /u02/app/oracle/oradata
mkdir -p /u03/app/oracle

mkfs.ext4 /dev/mapper/vg_oradata-lv_u01
mkfs.ext4 /dev/mapper/vg_oradata-lv_data
mkfs.ext4 /dev/mapper/vg_orareco-lv_reco

echo "/dev/mapper/vg_oradata-lv_u01 /u01 ext4 defaults 1 2" >> /etc/fstab
echo "/dev/mapper/vg_oradata-lv_data /u02/app/oracle/oradata ext4 defaults 1 2" >> /etc/fstab
echo "/dev/mapper/vg_orareco-lv_reco /u03/app/oracle ext4 defaults 1 2" >> /etc/fstab

mount -a

df -h
Filesystem                      Size  Used Avail Use% Mounted on
devtmpfs                         16G     0   16G   0% /dev
tmpfs                            16G     0   16G   0% /dev/shm
tmpfs                            16G  8.6M   16G   1% /run
tmpfs                            16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/ol-root              44G  1.4G   43G   4% /
/dev/vda1                      1014M  184M  831M  19% /boot
tmpfs                           3.2G     0  3.2G   0% /run/user/0
/dev/mapper/vg_oradata-lv_u01    30G   45M   28G   1% /u01
/dev/mapper/vg_oradata-lv_data  158G   61M  150G   1% /u02/app/oracle/oradata
/dev/mapper/vg_orareco-lv_reco   59G   53M   56G   1% /u03/app/oracle

Let’s copy the ISO file on the server for package setup. I will mount this ISO file, configure a local repository on this ISO, and install the preinstall package. Other packages will be needed for 11gR2:

mkdir /install
scp root@10.168.1.54:/mnt/nas/V1009690-01.iso /install/

 
mkdir /mnt/iso
mount -o loop /install/V1009690-01.iso /mnt/iso
rm -f /etc/yum.repos.d/*
vi /etc/yum.repos.d/local-oel7.repo
[OEL790]
name=Oracle Linux 7.9 x86_64
baseurl=file:///mnt/iso
gpgkey=file:///mnt/iso/RPM-GPG-KEY
gpgcheck=1
enabled=1

yum install oracle-database-preinstall-19c.x86_64
...

yum install compat-libstdc++-33-3.2.3-72.el7.x86_64
yum install gcc-4.8.5-44.0.3.el7.x86_64
yum install gcc-c++-4.8.5-44.0.3.el7.x86_64
yum install mlocate

Now I will do the 11gR2 setup using an image from the old system and the cloning method: I don’t want any changes regarding the binaries.

scp root@10.168.1.54:/mnt/nas/root_SRV-BD2_dbhome_11gR2.tgz /install/

chmod -R 755 /install/ 
chown -R oracle:oinstall /u01/
chown -R oracle:oinstall /u02/
chown -R oracle:oinstall /u03/

 
su - oracle
mkdir -p /u01/app/oracle/product/11.2.0.4/
mkdir -p /u01/app/oracle/network/admin
mkdir /u01/app/oracle/local
cd /u01/app/oracle/product/11.2.0.4/
tar xzf /install/root_SRV-BD2_dbhome_11gR2.tgz 
cd /u01/app/oracle/product/11.2.0.4/dbhome_1/clone/bin
/u01/app/oracle/product/11.2.0.4/dbhome_1/clone/bin/clone.pl ORACLE_BASE="/u01/app/oracle" ORACLE_HOME="/u01/app/oracle/product/11.2.0.4/dbhome_1" ORACLE_HOME_NAME=OraDB11gHome1 OSDBA_GROUP=dba OSOPER_GROUP=oper
...

exit

/u01/app/oraInventory/orainstRoot.sh
/u01/app/oracle/product/11.2.0.4/dbhome_1/root.sh

I will disable SELinux and the firewall, I don’t need them:

vi /etc/selinux/config
SELINUX=disabled
systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
systemctl stop firewalld
reboot

A relink of the binaries was needed:

su - oracle
export ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome_1
$ORACLE_HOME/bin/relink all

Now I need to configure a default listener:

su – oracle
$ORACLE_HOME/bin/netca -silent -responsefile /u01/app/oracle/product/11.2.0.4/dbhome_1/assistants/netca/netca.rsp

Parsing command line arguments:
    Parameter "silent" = true
    Parameter "responsefile" = /u01/app/oracle/product/11.2.0.4/dbhome_1/assistants/netca/netca.rsp
Done parsing command line arguments.
Oracle Net Services Configuration:
Profile configuration complete.
Oracle Net Listener Startup:
    Running Listener Control:
      /u01/app/oracle/product/11.2.0.4/dbhome_1/bin/lsnrctl start LISTENER
    Listener Control complete.
    Listener started successfully.
Listener configuration complete.
Oracle Net Services configuration successful. The exit code is 0

My system is now ready for “new” 11gR2 databases.

Next steps

The system is now deployed and ready to host a first database. Creating a database is done by using dbca or by restoring a database from a backup. Using a backup is definitely the best idea to keep the database as close as the source one. You may need to use db_file_name_convert to remap the old datafile structure to the new one. I would probably create a pfile from the source database, make the needed changes and start my new instance with this pfile.

Conclusion

Only consider this solution if no other one is possible. Remember that it comes without any support and without any performance guarantee. For sure, I wouldn’t recommend using this solution for a production database, and I definitely advise migrating all your old databases to 19c. Don’t forget that 11gR2 is more than 15-year old now, and you shouldn’t deploy it in 2023.

L’article Deploying 11.2.0.4 on a recent ODA est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/deploying-11-2-0-4-on-a-recent-oda/feed/ 4 DB Systems backup on ODA https://www.dbi-services.com/blog/db-systems-backup-on-oda/ https://www.dbi-services.com/blog/db-systems-backup-on-oda/#comments Tue, 20 Dec 2022 17:03:12 +0000 https://www.dbi-services.com/blog/?p=21153 How to backup DB Systems on Oracle Database Appliance

L’article DB Systems backup on ODA est apparu en premier sur dbi Blog.

]]> Introduction

All Oracle Database Appliances running recent versions of the patch (19.1x) are now able to run bare metal databases, virtualized databases or both. A virtualized database is actually a single database in a dedicated VM called a DB System. This is exactly what you can find in OCI, the Oracle public cloud. This feature is very convenient to isolate databases in different networks, and dedicate a limited amount of memory and CPU resources. You may think that these DB Systems have a kind of backup command proposed by odacli, but it does not exist for now.

odacli -h | grep dbsystem | sort | uniq
		create-dbsystem
		delete-dbsystem
		describe-dbsystem
		describe-dbsystem-image
		list-dbsystems
		modify-dbsystem
		start-dbsystem
		stop-dbsystem
	dbsystem-image:
	dbsystem:

Is there a solution to make simple backups of these DB Systems? Let’s try to find out.

DB Systems and storage

DB Systems use 2 kind of storages. A vdisk for the system itself, and ASM storage for data and recovery area. The ASM storage proposed in a DB System is nothing else than the ASM storage of your ODA, and it’s shared accross all VMs. ASM storage is only used for database files, and as you know you will backup your database using classic RMAN scripts or with odacli which rely on RMAN.

There is no tool to backup the vdisk, the vdisk being an hidden file on this filesystem: /u05/app/sharedrepo/hostname. It’s not an issue not having backups of this vdisk because you would normally be able to restore the RMAN backup of your database on another DB System. DB System are not supposed to be tuned manually. The only operation that is normally supported is to modify the DB System with modify-dbsystem, mainly to change its shape (resources allocation).

Most probably, nothing will never corrupt or destroy your DB System, but as you never know, it could be nice to do backups from time to time, just in case.

Identifying the vdisk

Let’s list the DB Systems on my ODA:

odacli list-dbsystems
Name                  Shape       Cores  Memory      GI version          DB version          Status           Created                   Updated
--------------------  ----------  -----  ----------  ------------------  ------------------  ---------------  ------------------------  ------------------------
srvdb01               odb2        2      16.00 GB    19.16.0.0.220719    19.16.0.0.220719    CONFIGURED       2022-11-04 15:03:28 CET   2022-11-04 15:29:51 CET

Finding the associated vdisk of this unique DB System is quite easy:

odacli describe-dbsystem -n srvdb01 | grep "VM image"
            VM image path:  /u05/app/sharedrepo/srvdb01/.ACFS/snaps/vm_x9622f3fdf/x9622f3fdf

My DB System is running, so the vdisk is continuously updated:

ls -lrth /u05/app/sharedrepo/srvdb01/.ACFS/snaps/vm_x9622f3fdf/x9622f3fdf
-rw-r--r-- 1 qemu qemu 53G Dec 20 14:54 /u05/app/sharedrepo/srvdb01/.ACFS/snaps/vm_x9622f3fdf/x9622f3fdf

sleep 60 ; ls -lrth /u05/app/sharedrepo/srvdb01/.ACFS/snaps/vm_x9622f3fdf/x9622f3fdf
-rw-r--r-- 1 qemu qemu 53G Dec 20 14:55 /u05/app/sharedrepo/srvdb01/.ACFS/snaps/vm_x9622f3fdf/x9622f3fdf

A clean backup will first need stopping the DB System:

odacli stop-dbsystem -n srvdb01
odacli describe-dbsystem -n srvdb01 | grep "Current State"
            Current State:  OFFLINE

Backing up the vdisk

Now let’s copy the vdisk on a backup location. In this example I will use my ACFS recovery area volume but in real life it will probably be a NFS share:

cp /u05/app/sharedrepo/srvdb01/.ACFS/snaps/vm_x9622f3fdf/x9622f3fdf /u03/app/oracle/`date +"%Y%m%d_%H%M"`_srvdb01_vdisk.bck

Let’s check if everything is OK with the copy:

md5sum /u03/app/oracle/20221220_1501_srvdb01_vdisk.bck
c00c5d1908eca913b5893c277540439b  /u03/app/oracle/20221220_1501_srvdb01_vdisk.bck

md5sum /u05/app/sharedrepo/srvdb01/.ACFS/snaps/vm_x9622f3fdf/x9622f3fdf
c00c5d1908eca913b5893c277540439b  /u05/app/sharedrepo/srvdb01/.ACFS/snaps/vm_x9622f3fdf/x9622f3fdf

Now I can start again my DB System:

odacli start-dbsystem -n srvdb01

odacli describe-dbsystem -n srvdb01 | grep "Current State"
            Current State:  ONLINE

Destroying my DB System

Destroying systems and databases is always easy! Let’s connect to the DB System using its private IP:

odacli describe-dbsystem -n srvdb01 | grep "ASM:"
                      ASM:  192.168.17.4    / 255.255.255.128 / ens4 / BRIDGE(privasm) VLAN(priv0.100)

ssh 192.168.17.4
FIPS mode initialized
root@192.168.17.4's password:
Last login: Tue Dec 20 14:32:50 2022 from 192.168.17.2
ps -ef | grep pmon
oracle   19650     1  0 15:31 ?        00:00:00 ora_pmon_VDBITST
root     25612 25327  0 15:33 pts/0    00:00:00 grep --color=auto pmon

I will create a table before destroying the system:

su - oracle
. oraenv <<< VDBITST
sqlplus / as sysdba
SQL> create table before_the_crash as select sysdate "THE_DATE" from dual;

Table created.

SQL> exit;

exit

My database VDBITST is running fine, but it will end soon.

Let’s destroy the / filesystem:

rm -rf /
rm: it is dangerous to operate recursively on '/'
rm: use --no-preserve-root to override this failsafe
rm -rf / --no-preserve-root
...
df -h
-bash: df: command not found
shutdown -h now
-bash: shutdown: command not found

exit
Connection to 192.168.17.4 closed by remote host.
Connection to 192.168.17.4 closed.

My DB System is not working correctly anymore. Let’s stop it from the ODA:

odacli stop-dbsystem -n srvdb01
odacli describe-dbsystem -n srvdb01 | grep "Current State"
            Current State:  ONLINE
sleep 30 ; odacli describe-dbsystem -n srvdb01 | grep "Current State"
            Current State:  ONLINE
sleep 30 ; odacli describe-dbsystem -n srvdb01 | grep "Current State"
            Current State:  ONLINE

odacli describe-job -i 7a450741-589a-4c6c-9f80-3888a52db91e

Job details
----------------------------------------------------------------
                     ID:  7a450741-589a-4c6c-9f80-3888a52db91e
            Description:  DB System srvdb01 stop
                 Status:  Running
                Created:  December 20, 2022 3:36:46 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Stop DB System                           December 20, 2022 3:36:46 PM CET    December 20, 2022 3:36:46 PM CET    Running

odacli describe-job -i 7a450741-589a-4c6c-9f80-3888a52db91e

Job details
----------------------------------------------------------------
                     ID:  7a450741-589a-4c6c-9f80-3888a52db91e
            Description:  DB System srvdb01 stop
                 Status:  Success
                Created:  December 20, 2022 3:36:46 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Stop DB System                           December 20, 2022 3:36:46 PM CET    December 20, 2022 3:46:50 PM CET    Success

It took 10 minutes to shut down because odacli didn’t managed to do a clean stop, obviously.

Restoring the DB System

Restoring the DB System is actually restoring the vdisk. So let’s copy back the clean vdisk image from my backup folder:

mv /u05/app/sharedrepo/srvdb01/.ACFS/snaps/vm_x9622f3fdf/x9622f3fdf /u05/app/sharedrepo/srvdb01/.ACFS/snaps/vm_x9622f3fdf/x9622f3fdf.old

cp /u03/app/oracle/20221220_1501_srvdb01_vdisk.bck /u05/app/sharedrepo/srvdb01/.ACFS/snaps/vm_x9622f3fdf/x9622f3fdf

Let’s then start again the DB System:

odacli start-dbsystem -n srvdb01
odacli describe-dbsystem -n srvdb01 | grep "Current State"
            Current State:  ONLINE

And finally, let’s connect to it and check if everything is OK:

ssh 192.168.17.4
FIPS mode initialized
root@192.168.17.4's password:
Last login: Tue Dec 20 14:32:50 2022 from 192.168.17.2
df -h
Filesystem                             Size  Used Avail Use% Mounted on
devtmpfs                               7.7G   16K  7.7G   1% /dev
tmpfs                                  7.8G     0  7.8G   0% /dev/shm
tmpfs                                  7.8G  8.7M  7.7G   1% /run
tmpfs                                  7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/VolGroupVm-LogVolRoot       30G  3.5G   25G  13% /
/dev/mapper/VolGroupVm-LogVolOpt        25G  5.6G   18G  24% /opt
/dev/mapper/VolGroupVm-LogVolU01        99G   26G   68G  28% /u01
/dev/vda1                              942M  113M  765M  13% /boot
192.168.17.2:/opt/oracle/oak/pkgrepos   50G   38G  9.1G  81% /opt/oracle/oak/pkgrepos
tmpfs                                  1.6G     0  1.6G   0% /run/user/0

ps -ef | grep pmon
oracle   17286     1  0 15:58 ?        00:00:00 ora_pmon_VDBITST
root     18790 14644  0 15:58 pts/0    00:00:00 grep --color=auto pmon

It looks fine.

What about my table?

Let’s check if the database is OK and if the table created before the crash but after the vdisk backup is still there:

su - oracle
. oraenv <<< VDBITST
sqlplus / as sysdba
SQL> select status from v$instance;

STATUS
------------
OPEN

SQL> create table back_to_life as select sysdate "THE_DATE" from dual;

Table created.

SQL> alter session set nls_date_format='YYYYMMDD HH24MI';

Session altered.

SQL> select the_date from back_to_life;

THE_DATE
-------------
20221220 1600

SQL> select the_date from before_the_crash;

THE_DATE
-------------
20221220 1536

My DB System is back, and my database is OK. Data created after the vdisk backup was done is still there as expected.

Conclusion

odacli may include a backup feature for the DB Systems in a future release. But for now, doing a cold backup of the vdisks from time to time is a nice solution.

L’article DB Systems backup on ODA est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/db-systems-backup-on-oda/feed/ 1 ODA version 19.17 is available: how to patch? https://www.dbi-services.com/blog/oda-version-19-17-is-available-how-to-patch/ https://www.dbi-services.com/blog/oda-version-19-17-is-available-how-to-patch/#comments Fri, 02 Dec 2022 17:45:01 +0000 https://www.dbi-services.com/blog/?p=20908 How to patch your ODA to 19.17

L’article ODA version 19.17 is available: how to patch? est apparu en premier sur dbi Blog.

]]> Introduction

Patch 19.17 is now available on Oracle Database Appliance. It’s time to test it.

What’s new?

This version brings latest PSUs to database and grid homes with their bug fixes, as usual. It also brings latest 21.8 databases but as DB Systems only, 21c being an innovation release. There are not that many new features and it’s not bad. No major OS upgrade, no breaking new feature, no big changes. At some points, most of us prefer stability and easy-to-apply patches.

The only new features I’ve read from the release note are:

  • Error correlation in the Browser User Interface (BUI), making a kind of crosscheck between odacli errors and log files from the various components. As you may know, odacli is just an interface on top of classic OS and Oracle tools, and it’s sometimes tough to find the log which refer to the error
  • Database patching can now be done at the database level, and not only at the DB home level, with oracle update-database
  • Enhanced patching when using Data Guard to limit downtime: I’m not so sure it will change something as it’s convenient to switch all the databases prior starting to patch

Which ODA is compatible with this 19.17?

The new ODAs X9-2S/L/HA and for sure X8, X7 and X6 series. X5-2HA is still on the compatibility list, so you can keep this old appliance up-to-date.

Is this patch a cumulative one?

This 19.17 can be applied on top of 19.13 or later. I just applied this new patch on an X7-2M running on 19.9 using 19.13 as an intermediate version.

Is there also a patch for my databases?

Only databases version 19c are now supported, and this is OK because this is the only one you should use now. Good bye 12.1. You may use Data Preserving Reprovisioning if you come from a very old version (<=18.8) and want to keep your 11g or 18c for example. I wrote a blog post on this feature a few months ago.

It’s not in the new feature list but it seems that it’s now possible to register an old DB clone without any problem. But I would recommend to only use 19c with this patch version. Using old databases is not supported, may not work properly with odacli, and you should be aware that 11g and 12cR1 are over now.

Download the patch and clone files

Download the patch and the corresponding clones to be able to apply the complete patch.

34753059 => the patch itself
30403673 => the GI clone needed for deploying newer GI version (mandatory)
30403662 => the DB clone for deploying new version of 19c

You don’t need the ISO file for patching, but I would recommend to download it (patch 30403643).

Be sure to choose the very latest 19.17 when downloading the clones, download link will first propose older versions.

Prepare the patching

Before running prepatch, please check these prerequisites:

  • filesystems have 20% available free space (does not concern acfs volumes)
  • additional rpms manually installed should be removed
  • revert profile scripts to default’s one (for grid and oracle users)
  • make sure you planned a generous downtime, 4 hours being the bare minimum for patching and troubleshooting. 1 day is never too much.

You should use odabr to make snapshots of the important filesystems prior patching. It’s fast and doesn’t cost anything. I would also recommend doing a backup of most important files: tnsnames.ora and listener.ora, database list, network configuration files, aso.

Version precheck

Start to check current version on all components:

odacli describe-component | grep -v ^$

System Version
---------------
19.13.0.0.0
System node Name
---------------
uns-oda2
Local System Version
---------------
19.13.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.13.0.0.0           up-to-date
GI                                        19.13.0.0.211019      up-to-date
DB {
[ OraDB19000_home3,OraDB19000_home4 ]     19.13.0.0.211019      up-to-date
}
DCSCONTROLLER                             19.13.0.0.0           up-to-date
DCSCLI                                    19.13.0.0.0           up-to-date
DCSAGENT                                  19.13.0.0.0           up-to-date
DCSADMIN                                  19.13.0.0.0           up-to-date
OS                                        7.9                   up-to-date
ILOM                                      5.0.2.24.r141466      up-to-date
BIOS                                      41100500              up-to-date
SHARED CONTROLLER FIRMWARE                QDV1RF32              up-to-date
LOCAL DISK FIRMWARE                       0121                  up-to-date
SHARED DISK FIRMWARE                      0121                  up-to-date
HMP                                       2.4.8.0.600           up-to-date

Once the patch will be registered in the ODA repository, the “Available Version” column will be updated with versions provided within the patch.

Prepararing the patch and updating the DCS tools

Copy the patch files on your ODA in a temp directory. On ODA X9-2 system disks are now smaller, so don’t hesitate to put the file on an nfs share or in the FRA acfs volume if you have acfs configured for your databases. Then unzip the files:

cd /u03/app/oracle
unzip -o p30403662_1917000_Linux-x86-64.zip  
unzip -o p30403673_1917000_Linux-x86-64.zip  
unzip -o p34753059_1917000_Linux-x86-64.zip

I first tried to register both patch and GI, but coming from 19.13 it doesn’t work:

odacli update-repository -f /u03/app/oracle/oda-sm-19.17.0.0.0-221126.1-server.zip,/u03/app/oracle/odacli-dcs-19.17.0.0.0-221029-GI-19.17.0.0.zip

DCS-10001:Internal error encountered: Cannot find the corresponding image for /u03/app/oracle/odacli-dcs-19.17.0.0.0-221029-GI-19.17.0.0.zip in img_metadata.

It will work if you come from a newer version, from 19.16 for example.

But it’s not a problem as you can first register the patch only, update the DCS components and then register the GI clone:

odacli update-repository -f /u03/app/oracle/oda-sm-19.17.0.0.0-221126.1-server.zip
odacli describe-job -i "dbacdf31-405f-4f08-acf2-21d78b66945a"

Job details
----------------------------------------------------------------
                     ID:  dbacdf31-405f-4f08-acf2-21d78b66945a
            Description:  Repository Update
                 Status:  Success
                Created:  December 2, 2022 9:55:39 AM CET
                Message:  /u03/app/oracle/oda-sm-19.17.0.0.0-221126.1-server.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle                             December 2, 2022 9:55:40 AM CET     December 2, 2022 9:56:09 AM CET     Success

odacli describe-component | grep -v ^$ 
System Version
---------------
19.13.0.0.0
System node Name
---------------
uns-oda2
Local System Version
---------------
19.13.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.13.0.0.0           19.17.0.0.0
GI                                        19.13.0.0.211019      19.17.0.0.221018
DB {
[ OraDB19000_home3,OraDB19000_home4 ]     19.13.0.0.211019      19.17.0.0.221018
}
DCSCONTROLLER                             19.13.0.0.0           19.17.0.0.0
DCSCLI                                    19.13.0.0.0           19.17.0.0.0
DCSAGENT                                  19.13.0.0.0           19.17.0.0.0
DCSADMIN                                  19.13.0.0.0           19.17.0.0.0
OS                                        7.9                   up-to-date
ILOM                                      5.0.2.24.r141466      5.1.0.23.r146986
BIOS                                      41100500              41120100
SHARED CONTROLLER FIRMWARE                QDV1RF32              QDV1RF35
LOCAL DISK FIRMWARE                       0121                  not-available
SHARED DISK FIRMWARE                      0121                  not-available
HMP                                       2.4.8.0.600           2.4.8.9.603

Patching from 19.13 will normally be easy as there are not that many changes.

odacli update-dcsadmin -v 19.17.0.0.0
sleep 60; odacli describe-job -i "1b7ddd74-6108-4480-8584-251932476f44"
Job details
----------------------------------------------------------------
                     ID:  1b7ddd74-6108-4480-8584-251932476f44
            Description:  DcsAdmin patching
                 Status:  Success
                Created:  December 2, 2022 9:57:42 AM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                December 2, 2022 9:57:42 AM CET     December 2, 2022 9:57:42 AM CET     Success
dcs-admin upgrade                        December 2, 2022 9:57:42 AM CET     December 2, 2022 9:57:49 AM CET     Success

 
odacli update-dcscomponents -v 19.17.0.0.0
This job is interactive
{
  "jobId" : "c3ed2039-82b4-45a4-b7d4-813da9b2aa1a",
  "status" : "Success",
  "message" : "Update-dcscomponents is successful on all the node(s):DCS-Agent shutdown is successful. MySQL upgrade is successful. Metadata migration is done before. Metadata schema update is done. dcsagent RPM upgrade is successful.  dcscli RPM upgrade is successful.  dcscontroller RPM upgrade is successful.  Successfully ran setupAgentAuth.sh zookeeper RPM upgrade is successful.  ",
  "reports" : null,
  "createTimestamp" : "December 02, 2022 10:03:37 AM CET",
  "description" : "Update-dcscomponents job completed and is not part of Agent job list",
  "updatedTime" : "December 02, 2022 10:05:16 AM CET"
}

odacli update-dcsagent -v 19.17.0.0.0
sleep 180; odacli describe-job -i "716a8259-1404-454e-90fb-cf59c14921d7"
Job details
----------------------------------------------------------------
                     ID:  716a8259-1404-454e-90fb-cf59c14921d7
            Description:  DcsAgent patching
                 Status:  Success
                Created:  December 2, 2022 10:06:21 AM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Stop DCS Admin                           December 2, 2022 10:06:22 AM CET    December 2, 2022 10:06:22 AM CET    Success
Generate mTLS certificates               December 2, 2022 10:06:22 AM CET    December 2, 2022 10:06:23 AM CET    Success
Exporting Public Keys                    December 2, 2022 10:06:23 AM CET    December 2, 2022 10:06:24 AM CET    Success
Creating Trust Store                     December 2, 2022 10:06:24 AM CET    December 2, 2022 10:06:26 AM CET    Success
Update config files                      December 2, 2022 10:06:26 AM CET    December 2, 2022 10:06:26 AM CET    Success
Restart DCS Admin                        December 2, 2022 10:06:26 AM CET    December 2, 2022 10:06:46 AM CET    Success
Dcs-agent upgrade  to version 19.17.0.0.0 December 2, 2022 10:06:46 AM CET    December 2, 2022 10:07:58 AM CET    Success
Update System version                    December 2, 2022 10:07:58 AM CET    December 2, 2022 10:07:58 AM CET    Success

odacli update-repository -f /u03/app/oracle/odacli-dcs-19.17.0.0.0-221029-GI-19.17.0.0.zip
sleep 60; odacli describe-job -i "4f81cf82-fed7-47a0-a6c3-69d3f9193e65"

Job details
----------------------------------------------------------------
                     ID:  4f81cf82-fed7-47a0-a6c3-69d3f9193e65
            Description:  Repository Update
                 Status:  Success
                Created:  December 2, 2022 10:09:44 AM CET
                Message:  /u03/app/oracle/odacli-dcs-19.17.0.0.0-221029-GI-19.17.0.0.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle                             December 2, 2022 10:09:45 AM CET    December 2, 2022 10:10:21 AM CET    Success

Prepatching report

Let’s do the prepatching test:

odacli create-prepatchreport -s -v 19.17.0.0.0
sleep 600 ; odacli describe-prepatchreport -i bb50b38f-55be-424a-bdf9-7d70206a61e1

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  bb50b38f-55be-424a-bdf9-7d70206a61e1
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER, SERVER]
                 Status:  FAILED
                Created:  December 2, 2022 10:12:55 AM CET
                 Result:  One or more pre-checks failed for [GI]

Node Name
---------------
uns-oda2

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
 
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Failed    Lock on central inventory detected:
                                          /u01/app/oraInventory/locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Validate space for clones       Success   Clones volume is already created
volume
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate command execution      Success   Validated command execution

On my configuration it didn’t work but this failure is a known issue:

rm -rf /u01/app/oraInventory/locks

odacli create-prepatchreport -s -v 19.17.0.0.0
sleep 600 ; odacli describe-prepatchreport -i d334c952-7bd8-4dc1-bf44-ee9247f7acc2

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  d334c952-7bd8-4dc1-bf44-ee9247f7acc2
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER, SERVER]
                 Status:  SUCCESS
                Created:  December 2, 2022 10:23:01 AM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
uns-oda2

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
 
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Validate space for clones       Success   Clones volume is already created
volume
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate command execution      Success   Validated command execution

Everything is OK to start patching.

Patching server and GI

Let’s start the update-server:

odacli update-server -v 19.17.0.0.0
odacli describe-job -i "6e32cf55-54b5-41fc-af07-0cb773e68132"
Job details
----------------------------------------------------------------
                     ID:  6e32cf55-54b5-41fc-af07-0cb773e68132
            Description:  Server Patching
                 Status:  Success
                Created:  December 2, 2022 10:35:33 AM CET
                Message:  Successfully patched GI with RHP

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validating GI user metadata              December 2, 2022 10:35:46 AM CET    December 2, 2022 10:35:46 AM CET    Success
Validate ILOM server reachable           December 2, 2022 10:35:46 AM CET    December 2, 2022 10:35:46 AM CET    Success
Validate DCS Admin mTLS setup            December 2, 2022 10:35:46 AM CET    December 2, 2022 10:35:46 AM CET    Success
Configure export clones resource         December 2, 2022 10:35:47 AM CET    December 2, 2022 10:35:48 AM CET    Success
Creating repositories using yum          December 2, 2022 10:35:48 AM CET    December 2, 2022 10:35:52 AM CET    Success
Updating YumPluginVersionLock rpm        December 2, 2022 10:35:52 AM CET    December 2, 2022 10:35:52 AM CET    Success
Applying OS Patches                      December 2, 2022 10:35:52 AM CET    December 2, 2022 10:46:25 AM CET    Success
Creating repositories using yum          December 2, 2022 10:46:26 AM CET    December 2, 2022 10:46:26 AM CET    Success
Applying HMP Patches                     December 2, 2022 10:46:26 AM CET    December 2, 2022 10:46:46 AM CET    Success
Patch location validation                December 2, 2022 10:46:46 AM CET    December 2, 2022 10:46:46 AM CET    Success
Oda-hw-mgmt upgrade                      December 2, 2022 10:46:47 AM CET    December 2, 2022 10:47:21 AM CET    Success
OSS Patching                             December 2, 2022 10:47:21 AM CET    December 2, 2022 10:47:21 AM CET    Success
Applying Firmware Disk Patches           December 2, 2022 10:47:21 AM CET    December 2, 2022 10:47:23 AM CET    Success
Applying Firmware Controller Patches     December 2, 2022 10:47:24 AM CET    December 2, 2022 10:48:12 AM CET    Success
Checking Ilom patch Version              December 2, 2022 10:48:12 AM CET    December 2, 2022 10:48:13 AM CET    Success
Patch location validation                December 2, 2022 10:48:13 AM CET    December 2, 2022 10:48:13 AM CET    Success
Save password in Wallet                  December 2, 2022 10:48:13 AM CET    December 2, 2022 10:48:13 AM CET    Success
Disabling IPMI v2                        December 2, 2022 10:48:13 AM CET    December 2, 2022 10:48:14 AM CET    Success
Apply Ilom patch                         December 2, 2022 10:48:14 AM CET    December 2, 2022 10:57:11 AM CET    Success
Copying Flash Bios to Temp location      December 2, 2022 10:57:11 AM CET    December 2, 2022 10:57:11 AM CET    Success
Starting the clusterware                 December 2, 2022 10:57:11 AM CET    December 2, 2022 10:58:54 AM CET    Success
Registering image                        December 2, 2022 10:58:54 AM CET    December 2, 2022 10:58:54 AM CET    Success
Registering working copy                 December 2, 2022 10:58:54 AM CET    December 2, 2022 10:58:54 AM CET    Success
Registering image                        December 2, 2022 10:58:54 AM CET    December 2, 2022 10:58:54 AM CET    Success
Creating GI home directories             December 2, 2022 10:58:54 AM CET    December 2, 2022 10:58:54 AM CET    Success
Extract GI clone                         December 2, 2022 10:58:54 AM CET    December 2, 2022 10:58:55 AM CET    Success
Provisioning Software Only GI with RHP   December 2, 2022 10:58:55 AM CET    December 2, 2022 10:58:55 AM CET    Success
Patch GI with RHP                        December 2, 2022 10:58:55 AM CET    December 2, 2022 11:05:10 AM CET    Success
Updating GIHome version                  December 2, 2022 11:05:10 AM CET    December 2, 2022 11:05:14 AM CET    Success
Validate GI availability                 December 2, 2022 11:05:52 AM CET    December 2, 2022 11:05:52 AM CET    Success
Patch KVM CRS type                       December 2, 2022 11:05:52 AM CET    December 2, 2022 11:05:52 AM CET    Success
Patch VM vDisks CRS dependencies         December 2, 2022 11:05:52 AM CET    December 2, 2022 11:05:52 AM CET    Success
Patch DB System domain config            December 2, 2022 11:05:53 AM CET    December 2, 2022 11:05:53 AM CET    Success
Update System version                    December 2, 2022 11:05:53 AM CET    December 2, 2022 11:05:53 AM CET    Success
Cleanup JRE Home                         December 2, 2022 11:05:53 AM CET    December 2, 2022 11:05:53 AM CET    Success
Add SYSNAME in Env                       December 2, 2022 11:05:53 AM CET    December 2, 2022 11:05:53 AM CET    Success
Starting the clusterware                 December 2, 2022 11:05:53 AM CET    December 2, 2022 11:05:53 AM CET    Success
Setting ACL for disk groups              December 2, 2022 11:05:53 AM CET    December 2, 2022 11:05:57 AM CET    Success
Enable LKCE                              December 2, 2022 11:07:50 AM CET    December 2, 2022 11:10:37 AM CET    Success
Update previous workarounds              December 2, 2022 11:10:37 AM CET    December 2, 2022 11:10:37 AM CET    Success
Generating and saving BOM                December 2, 2022 11:10:37 AM CET    December 2, 2022 11:12:32 AM CET    Success
PreRebootNode Actions                    December 2, 2022 11:12:32 AM CET    December 2, 2022 11:13:19 AM CET    Success
Reboot Ilom                              December 2, 2022 11:13:19 AM CET    December 2, 2022 11:13:19 AM CET    Success

Server reboots 5 minutes after the patch ends. On this X7-2M server patching lasted 40 minutes.

Checking filesystems told me that there is a problem on the /boot. It’s quite common on an ODA where multiple patches were applied. If you have this problem, please identify the old kernels and remove the rpms packages:

df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        474M  451M     0 100% /boot

uname -a
Linux uns-oda2 4.14.35-2047.518.4.1.el7uek.x86_64 #2 SMP Tue Oct 18 18:08:21 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux
rpm -qa | grep kernel-uek-4
kernel-uek-4.14.35-2047.508.3.2.el7uek.x86_64
kernel-uek-4.14.35-2025.400.9.el7uek.x86_64
kernel-uek-4.14.35-2047.518.4.1.el7uek.x86_64

rpm -e kernel-uek-4.14.35-2025.400.9.el7uek.x86_64

df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        474M  331M  119M  74% /boot

Patching the storage

Patching the storage is only needed if describe-component tells you that you’re not up-to-date. On my X7-2M I had to patch:

odacli update-storage -v 19.17.0.0.0
odacli describe-job -i "93199b6d-b400-43cf-812f-20ffaf59d704"
Job details
----------------------------------------------------------------
                     ID:  93199b6d-b400-43cf-812f-20ffaf59d704
            Description:  Storage Firmware Patching
                 Status:  Success
                Created:  December 2, 2022 11:29:47 AM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Applying Firmware Disk Patches           December 2, 2022 11:29:53 AM CET    December 2, 2022 11:29:55 AM CET    Success
Applying Firmware Controller Patches     December 2, 2022 11:29:56 AM CET    December 2, 2022 11:37:21 AM CET    Success
Generating and saving BOM                December 2, 2022 11:37:21 AM CET    December 2, 2022 11:38:28 AM CET    Success
PreRebootNode Actions                    December 2, 2022 11:38:28 AM CET    December 2, 2022 11:38:28 AM CET    Success
Reboot Ilom                              December 2, 2022 11:38:28 AM CET    December 2, 2022 11:38:28 AM CET    Success

I never encountered troubles during storage patching, so it should be fine.

Patching the DB homes

Time for patching the DB homes depends on the number of DB homes and number of databases. In this example, I will apply the patch on my two 19c DB homes:

odacli list-dbhomes
ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
f33e91e6-ac8c-48b1-a599-2e32ddf46b30     OraDB19000_home3     19.13.0.0.211019                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_3 CONFIGURED
804165be-d58e-44c6-b51d-c2729a99b8e9     OraDB19000_home4     19.13.0.0.211019                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4 CONFIGURED

The DB clone is needed:

cd /u03/app/oracle/
unzip -o p30403662_1917000_Linux-x86-64.zip
odacli update-repository -f /u03/app/oracle/odacli-dcs-19.17.0.0.0-221029-DB-19.17.0.0.zip
odacli describe-job -i "5ab4fcaa-2452-4f43-b329-cc3df03abbf7"

Job details
----------------------------------------------------------------
                     ID:  5ab4fcaa-2452-4f43-b329-cc3df03abbf7
            Description:  Repository Update
                 Status:  Success
                Created:  December 2, 2022 11:50:02 AM CET
                Message:  /u03/app/oracle/odacli-dcs-19.17.0.0.0-221029-DB-19.17.0.0.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle                             December 2, 2022 11:50:02 AM CET    December 2, 2022 11:50:38 AM CET    Success

A prepatching is also needed here:

odacli create-prepatchreport -d -i f33e91e6-ac8c-48b1-a599-2e32ddf46b30 -v 19.17.0.0.0
odacli describe-prepatchreport -i 4142cc7e-edbf-4aa2-9db1-cf5f2ea2ab94

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  4142cc7e-edbf-4aa2-9db1-cf5f2ea2ab94
            Description:  Patch pre-checks for [DB, ORACHKDB]: DbHome is OraDB19000_home3
                 Status:  FAILED
                Created:  December 2, 2022 11:55:08 AM CET
                 Result:  One or more pre-checks failed for [ORACHK]

Node Name
---------------
uns-oda2

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__DB__
Validate DB Home ID             Success   Validated DB Home ID:
                                          f33e91e6-ac8c-48b1-a599-2e32ddf46b30
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate minimum agent version  Success   Validated minimum agent version
Is GI upgraded                  Success   Validated GI is upgraded
Validate available space for    Success   Validated free space required under
db                                        /u01
Validate dbHomesOnACFS          Success   User has configured disk group for
configured                                Database homes on ACFS
Validate Oracle base            Success   Successfully validated Oracle Base
Is DB clone available           Success   Successfully validated clone file
                                          exists
Evaluate DBHome patching with   Success   Successfully validated updating
RHP                                       dbhome with RHP.  and local patching
                                          is possible
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Failed    ORAchk validation failed: .
Validate command execution      Success   Validated command execution
Check for parameter             Failed    AHF-3744: Database parameter
global_names                              GLOBAL_NAMES is not set to
                                          recommended value
Check for parameter             Failed    AHF-3744: Database parameter
global_names                              GLOBAL_NAMES is not set to
                                          recommended value
Check for parameter             Failed    AHF-3744: Database parameter
global_names                              GLOBAL_NAMES is not set to
                                          recommended value

I don’t care about Orachk recommendations on my databases because it has been set like that. I will apply the patch on this DB home with the force option:

odacli update-dbhome -i f33e91e6-ac8c-48b1-a599-2e32ddf46b30 -v 19.17.0.0.0 -f
sleep 600 ; odacli describe-job -i "0e4a424b-9892-46d9-90d8-ebb61737ef03"

Job details
----------------------------------------------------------------
                     ID:  0e4a424b-9892-46d9-90d8-ebb61737ef03
            Description:  DB Home Patching: Home Id is f33e91e6-ac8c-48b1-a599-2e32ddf46b30
                 Status:  Success
                Created:  December 2, 2022 12:11:31 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Creating wallet for DB Client            December 2, 2022 12:12:18 PM CET    December 2, 2022 12:12:18 PM CET    Success
Patch databases by RHP                   December 2, 2022 12:12:18 PM CET    December 2, 2022 12:18:34 PM CET    Success
Updating database metadata               December 2, 2022 12:18:34 PM CET    December 2, 2022 12:18:34 PM CET    Success
Set log_archive_dest for Database        December 2, 2022 12:18:34 PM CET    December 2, 2022 12:18:37 PM CET    Success
Patch databases by RHP                   December 2, 2022 12:18:37 PM CET    December 2, 2022 12:20:09 PM CET    Success
Updating database metadata               December 2, 2022 12:20:09 PM CET    December 2, 2022 12:20:09 PM CET    Success
Set log_archive_dest for Database        December 2, 2022 12:20:09 PM CET    December 2, 2022 12:20:13 PM CET    Success
Patch databases by RHP                   December 2, 2022 12:20:13 PM CET    December 2, 2022 12:25:58 PM CET    Success
Updating database metadata               December 2, 2022 12:25:58 PM CET    December 2, 2022 12:25:58 PM CET    Success
Set log_archive_dest for Database        December 2, 2022 12:25:58 PM CET    December 2, 2022 12:26:00 PM CET    Success
Update System version                    December 2, 2022 12:26:00 PM CET    December 2, 2022 12:26:00 PM CET    Success
Generating and saving BOM                December 2, 2022 12:26:00 PM CET    December 2, 2022 12:28:03 PM CET    Success
TDE parameter update                     December 2, 2022 12:28:39 PM CET    December 2, 2022 12:28:39 PM CET    Success

It’s the exact same procedure for the second DB home.

New DB homes have been created and my databases are now linked to these new ones:

odacli list-dbhomes
ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
f33e91e6-ac8c-48b1-a599-2e32ddf46b30     OraDB19000_home3     19.13.0.0.211019                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_3 CONFIGURED
804165be-d58e-44c6-b51d-c2729a99b8e9     OraDB19000_home4     19.13.0.0.211019                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4 CONFIGURED
7741bb27-43cc-484c-9525-5f854262c33d     OraDB19000_home5     19.17.0.0.221018                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5 CONFIGURED
cacbc994-04ae-4f40-ae96-a5b57bb3d89a     OraDB19000_home6     19.17.0.0.221018                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_6 CONFIGURED

odacli list-databases
ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
cbc31d02-5507-4e90-97b4-42a65bb2e2b7     PCGO_LCDF  SI       19.17.0.0.221018     false      OLTP     odb1     ACFS       CONFIGURED   7741bb27-43cc-484c-9525-5f854262c33d
118c79a4-c5cc-463c-ae29-d92208f502a7     PCSGO_LCDF SI       19.17.0.0.221018     false      OLTP     odb1     ACFS       CONFIGURED   cacbc994-04ae-4f40-ae96-a5b57bb3d89a
4892fc73-e89a-4fc4-8fac-c4d4f91ba64d     TCGO       SI       19.17.0.0.221018     false      OLTP     odb1     ACFS       CONFIGURED   7741bb27-43cc-484c-9525-5f854262c33d
ef0cc1fe-86b8-4f6d-9abe-a619147337ca     TCSGO      SI       19.17.0.0.221018     false      OLTP     odb1     ACFS       CONFIGURED   7741bb27-43cc-484c-9525-5f854262c33d

The old DB homes can now be safely removed:

odacli delete-dbhome -i f33e91e6-ac8c-48b1-a599-2e32ddf46b30
sleep 60 ;  odacli delete-dbhome -i 804165be-d58e-44c6-b51d-c2729a99b8e9
sleep 120 ; odacli list-jobs | tail -n 3
212c0662-a062-40c6-925b-3b915effa23d     Database Home OraDB19000_home3 Deletion with id f33e91e6-ac8c-48b1-a599-2e32ddf46b30 December 2, 2022 1:01:09 PM CET     Success
a64b32ac-9cad-4f23-acec-66077a5f856d     Database Home OraDB19000_home4 Deletion with id 804165be-d58e-44c6-b51d-c2729a99b8e9 December 2, 2022 1:01:41 PM CET     Success

Final checks

Let’s get the final versions:

odacli describe-component | grep -v ^$
System Version
---------------
19.17.0.0.0
System node Name
---------------
uns-oda2
Local System Version
---------------
19.17.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.17.0.0.0           up-to-date
GI
                                          19.17.0.0.221018      up-to-date
DB {
[OraDB19000_home5 [PCGO_LCDF,TCGO,
TCSGO]]                                   19.17.0.0.221018      up-to-date
[OraDB19000_home6 [PCSGO_LCDF]]
                                          19.17.0.0.221018      up-to-date
}
DCSCONTROLLER
                                          19.17.0.0.0           up-to-date
 
DCSCLI
                                          19.17.0.0.0           up-to-date
DCSAGENT
                                          19.17.0.0.0           up-to-date
DCSADMIN
                                          19.17.0.0.0           up-to-date
OS
                                          7.9                   up-to-date
ILOM
                                          5.1.0.23.r146986      up-to-date
BIOS
                                          41120100              up-to-date
LOCAL CONTROLLER FIRMWARE {
[c6]
                                          80000690              up-to-date
[c7]
                                          214.2.271.9           up-to-date
}
SHARED CONTROLLER FIRMWARE
                                          QDV1RF35              up-to-date
LOCAL DISK FIRMWARE
                                          N2010121              up-to-date
SHARED DISK FIRMWARE
                                          N2010121              up-to-date
HMP
                                          2.4.8.9.603           up-to-date

Everything is fine. I’m quiet for 2-3 months.

Cleanse the old patches

The old patches will never be used again, so don’t forget to remove them from the repository if your ODA has already been patched:

odacli cleanup-patchrepo -cl -v 19.13.0.0.0

odacli describe-job -i "7e1506b9-5778-47ed-84eb-65d976819c80"
Job details
----------------------------------------------------------------
                     ID:  7e1506b9-5778-47ed-84eb-65d976819c80
            Description:  Cleanup patchrepos
                 Status:  Success
                Created:  December 2, 2022 12:56:14 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Cleanup Repository                       December 2, 2022 12:56:14 PM CET    December 2, 2022 12:56:14 PM CET    Success
Cleanup JRE Home                         December 2, 2022 12:56:15 PM CET    December 2, 2022 12:56:15 PM CET    Success
Cleanup old ASR rpm                      December 2, 2022 12:56:15 PM CET    December 2, 2022 12:56:15 PM CET    Success

Put back your own settings

Once everything is OK, don’t forget to put back your settings:

  • add your additional rpms manually if needed
  • put back your profile scripts for grid and oracle users
  • …

Patching DB Systems

If you use DB Systems on your ODA, meaning that some of your databases are running in dedicated VMs, you will need to apply the patch inside each DB System. I haven’t tried on my X7-2M because it only runs bare metal databases, but you will need to list your DB Systems, connect to each one, and apply the server and database patches:

odacli list-dbsystems
ssh ...
odacli update-server -v 19.17.0.0.0
odacli update-dbhome -i ... -v 19.17.0.0.0

Conclusion

This release is easy to apply coming from 19.13 and also probably from newer versions. Future releases may include a major OS upgrade and 23c sooner or later.

L’article ODA version 19.17 is available: how to patch? est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/oda-version-19-17-is-available-how-to-patch/feed/ 2 Patch 19.14 is out for ODA: new features and how to apply https://www.dbi-services.com/blog/patch-19-14-is-out-for-oda-new-features-and-how-to-apply/ https://www.dbi-services.com/blog/patch-19-14-is-out-for-oda-new-features-and-how-to-apply/#respond Tue, 01 Mar 2022 16:12:28 +0000 https://www.dbi-services.com/blog/patch-19-14-is-out-for-oda-new-features-and-how-to-apply/ Introduction Patch 19.14 is now available on Oracle Database Appliance. It’s time to test it. What’s new? This version brings January’s PSU to database and grid homes with their bug fixes, as usual. It also brings latest 21.5 but only for DB Systems (21c being an innovation release). Don’t expect too many new features for […]

L’article Patch 19.14 is out for ODA: new features and how to apply est apparu en premier sur dbi Blog.

]]> Introduction

Patch 19.14 is now available on Oracle Database Appliance. It’s time to test it.

What’s new?

This version brings January’s PSU to database and grid homes with their bug fixes, as usual. It also brings latest 21.5 but only for DB Systems (21c being an innovation release). Don’t expect too many new features for this release (actually, I don’t see anything new), it’s much more a bundle of bug fixes and finally it’s what I like.

Most important new features were brought by previous 19.13: multi-user mode for odacli tasks and parameter repository for bare metal instances.

Which ODA is compatible with this 19.14?

ODA X8 is becoming quite old now but no X9 has been announced yet. So the X8 in its 3 flavors (S, M, HA) is compatible with this patch, so are X7, X6 and X5-2HA (probably one of the latest patch for this one).

Is this patch a cumulative one?

This 19.14 can be applied on top of 19.10 or later. If you’re using older version like 19.6 or 19.8, applying 19.10 is needed prior 19.14.

As usual, don’t hesitate to consider reimaging instead of patching if 3 or more patches are needed to reach 19.14.

Is there also a patch for my databases?

If you’re using 12.1, 12.2 or 19c, these databases can be patched. 11gR2 and 18c are no more supported on ODA, meaning that your databases will continue working but you should definitely consider an upgrade. If you choose to reimage to this latest version, you will not find any 18c or 11gR2 DB homes to deploy.

Download the patch and clone files

Download the patch and the corresponding clones to be able to apply the complete patch.

  • 33702951 => the patch itself
  • 30403673 => the GI clone needed for deploying new version (mandatory)
  • 30403662 => the DB clone for deploying new version of 19c (if you use 19c databases)
  • 27119402 => the DB clone for deploying new version of 12.2 (if you use 12.2 databases)
  • 23494992 => the DB clone for deploying new version of 12.1 (if you use 12.1 databases)

Be sure to choose the very latest 19.14 when downloading the clones, download link will first propose older versions.

Prepare the patching

A pre-patch is now provided and needed before applying an ODA patch, but prior running it, please check these prerequisites:

  • filesystems have 20% available free space (does not concern acfs volumes)
  • additional rpms manually installed should be removed
  • revert profile scripts to default’s one (concerns grid and oracle users)
  • make sure you can afford longer than planned downtime, 4 hours being the bare minimum for patching and troubleshooting. 1 day is never too much

You can use odabr to backup your filesystems to snapshot or to nfs, or simply backup all your important files to a nfs share in case patching would fail.

Version precheck

Start to check current version on all components:

odacli describe-component | grep -v ^$
System Version
---------------
19.13.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.13.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.13.0.0.0           up-to-date
GI                                        19.13.0.0.211019      up-to-date
DB                                        19.13.0.0.211019      up-to-date
DCSCONTROLLER                             19.13.0.0.0           up-to-date
DCSCLI                                    19.13.0.0.0           up-to-date
DCSAGENT                                  19.13.0.0.0           up-to-date
DCSADMIN                                  19.13.0.0.0           up-to-date
OS                                        7.9                   up-to-date
ILOM                                      5.0.2.24.r141466      up-to-date
BIOS                                      52050300              up-to-date
SHARED CONTROLLER FIRMWARE                VDV1RL04              up-to-date
LOCAL DISK FIRMWARE                       1132                  up-to-date
SHARED DISK FIRMWARE                      1132                  up-to-date
HMP                                       2.4.8.0.600           up-to-date

Once the patch will be registered in the ODA repository, the “Available Version” column will be fed with versions provided within the patch.

Patching from 19.13 will normally be easy.

Prepararing the patch files and updating the DCS tools

Copy the patch files on your ODA in a temp directory. Then unzip the files:

cd /opt/dbi/
for f in p*1914000*.zip; do unzip -n $f; done
Archive:  p30403662_1914000_Linux-x86-64.zip
 extracting: odacli-dcs-19.14.0.0.0-220215-DB-19.14.0.0.zip
Archive:  p30403673_1914000_Linux-x86-64.zip
 extracting: odacli-dcs-19.14.0.0.0-220215-GI-19.14.0.0.zip
Archive:  p33702951_1914000_Linux-x86-64.zip
 extracting: oda-sm-19.14.0.0.0-220215-server.zip

rm -rf p*1914000*.zip

Register the patch in the repository:

odacli update-repository -f /opt/dbi/oda-sm-19.14.0.0.0-220215-server.zip

odacli describe-component | grep -v ^$
System Version
---------------
19.13.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.13.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.13.0.0.0           19.14.0.0.0
GI                                        19.13.0.0.211019      19.14.0.0.220118
DB                                        19.13.0.0.211019      19.14.0.0.220118
DCSCONTROLLER                             19.13.0.0.0           19.14.0.0.0
DCSCLI                                    19.13.0.0.0           19.14.0.0.0
DCSAGENT                                  19.13.0.0.0           19.14.0.0.0
DCSADMIN                                  19.13.0.0.0           19.14.0.0.0
OS                                        7.9                   up-to-date
ILOM                                      5.0.2.24.r141466      up-to-date
BIOS                                      52050300              up-to-date
SHARED CONTROLLER FIRMWARE                VDV1RL04              up-to-date
LOCAL DISK FIRMWARE                       1132                  up-to-date
SHARED DISK FIRMWARE                      1132                  up-to-date
HMP                                       2.4.8.0.600           up-to-date

Update the DCS tooling of your ODA:

odacli update-dcsadmin -v 19.14.0.0.0
sleep 30;  odacli update-dcscomponents -v 19.14.0.0.0
odacli update-dcsagent -v 19.14.0.0.0

Note that updating the DCS components is not done through a job:

sleep 180; odacli list-jobs | head -n 3;  odacli list-jobs | tail -n 4
ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
0526db5b-69cd-42c6-b8af-93d34c7c261f     Repository Update                                                           March 1, 2022 9:31:20 AM CET        Success
bdf6a313-2e0c-4359-adb6-0e7606c38735     DcsAdmin patching                                                           March 1, 2022 9:32:52 AM CET        Success
b9905530-d1ad-41ec-9b52-94f62728d4b5     DcsAgent patching                                                           March 1, 2022 9:34:49 AM CET        Success

Now you can register GI and DB clones:

odacli update-repository -f /opt/dbi/odacli-dcs-19.14.0.0.0-220215-GI-19.14.0.0.zip
sleep 50; odacli update-repository -f /opt/dbi/odacli-dcs-19.14.0.0.0-220215-DB-19.14.0.0.zip

sleep 50; odacli list-jobs | head -n 3;  odacli list-jobs | tail -n 3
ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
d72451bf-c0da-42b8-8fd9-72d45af5aabc     Repository Update                                                           March 1, 2022 9:39:00 AM CET        Success
b94fcecc-549c-4011-b239-022a88aa68e2     Repository Update                                                           March 1, 2022 9:40:04 AM CET        Success

odacli describe-component | grep -v ^$
System Version
---------------
19.14.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.14.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.13.0.0.0           19.14.0.0.0
GI
                                          19.13.0.0.211019      19.14.0.0.220118
DB
                                          19.13.0.0.211019      19.14.0.0.220118
DCSCONTROLLER
                                          19.14.0.0.0           up-to-date
DCSCLI
                                          19.14.0.0.0           up-to-date
DCSAGENT
                                          19.14.0.0.0           up-to-date
DCSADMIN
                                          19.14.0.0.0           up-to-date
OS
                                          7.9                   up-to-date
ILOM
                                          5.0.2.24.r141466      up-to-date
BIOS
                                          52050300              up-to-date
SHARED CONTROLLER FIRMWARE
                                          VDV1RL04              up-to-date
LOCAL DISK FIRMWARE
                                          1132                  up-to-date
SHARED DISK FIRMWARE
                                          1132                  up-to-date
HMP
                                          2.4.8.0.600           up-to-date

This update will be limited to Oracle software.

Pre-patching report

Let’s check if patching has the green light:

odacli create-prepatchreport -s -v 19.14.0.0.0

odacli describe-prepatchreport -i 71867a62-37aa-4b95-a841-893977038cca

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  71867a62-37aa-4b95-a841-893977038cca
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
                 Status:  SUCCESS
                Created:  March 1, 2022 9:52:45 AM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
dbi-oda-x8

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

Everything is OK to start patching.

Patching infrastructure and GI

Let’s start the update-server:

odacli update-server -v 19.14.0.0.0
odacli describe-job -i 8e77a48d-672e-4b81-a6dc-6e396c32e99c

Job details
----------------------------------------------------------------
                     ID:  8e77a48d-672e-4b81-a6dc-6e396c32e99c
            Description:  Server Patching
                 Status:  Success
                Created:  March 1, 2022 10:11:30 AM CET
                Message:  Successfully patched GI with RHP

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validating GI user metadata              March 1, 2022 10:11:42 AM CET       March 1, 2022 10:11:42 AM CET       Success
Validate ILOM server reachable           March 1, 2022 10:11:42 AM CET       March 1, 2022 10:11:42 AM CET       Success
Configure export clones resource         March 1, 2022 10:11:43 AM CET       March 1, 2022 10:11:44 AM CET       Success
Creating repositories using yum          March 1, 2022 10:11:44 AM CET       March 1, 2022 10:11:46 AM CET       Success
Updating YumPluginVersionLock rpm        March 1, 2022 10:11:46 AM CET       March 1, 2022 10:11:47 AM CET       Success
Applying OS Patches                      March 1, 2022 10:11:47 AM CET       March 1, 2022 10:18:26 AM CET       Success
Creating repositories using yum          March 1, 2022 10:18:27 AM CET       March 1, 2022 10:18:27 AM CET       Success
Applying HMP Patches                     March 1, 2022 10:18:27 AM CET       March 1, 2022 10:18:28 AM CET       Success
Patch location validation                March 1, 2022 10:18:28 AM CET       March 1, 2022 10:18:28 AM CET       Success
oda-hw-mgmt upgrade                      March 1, 2022 10:18:28 AM CET       March 1, 2022 10:18:57 AM CET       Success
OSS Patching                             March 1, 2022 10:18:57 AM CET       March 1, 2022 10:18:57 AM CET       Success
Applying Firmware Disk Patches           March 1, 2022 10:18:57 AM CET       March 1, 2022 10:19:01 AM CET       Success
Applying Firmware Controller Patches     March 1, 2022 10:19:01 AM CET       March 1, 2022 10:19:05 AM CET       Success
Checking Ilom patch Version              March 1, 2022 10:19:05 AM CET       March 1, 2022 10:19:05 AM CET       Success
Patch location validation                March 1, 2022 10:19:05 AM CET       March 1, 2022 10:19:05 AM CET       Success
Save password in Wallet                  March 1, 2022 10:19:05 AM CET       March 1, 2022 10:19:05 AM CET       Success
Disabling IPMI v2                        March 1, 2022 10:19:06 AM CET       March 1, 2022 10:19:06 AM CET       Success
Apply Ilom patch                         March 1, 2022 10:19:06 AM CET       March 1, 2022 10:19:06 AM CET       Success
Copying Flash Bios to Temp location      March 1, 2022 10:19:06 AM CET       March 1, 2022 10:19:06 AM CET       Success
Starting the clusterware                 March 1, 2022 10:19:06 AM CET       March 1, 2022 10:20:48 AM CET       Success
registering image                        March 1, 2022 10:20:49 AM CET       March 1, 2022 10:20:49 AM CET       Success
registering working copy                 March 1, 2022 10:20:49 AM CET       March 1, 2022 10:20:49 AM CET       Success
registering image                        March 1, 2022 10:20:49 AM CET       March 1, 2022 10:20:49 AM CET       Success
Creating GI home directories             March 1, 2022 10:20:49 AM CET       March 1, 2022 10:20:49 AM CET       Success
Extract GI clone                         March 1, 2022 10:20:49 AM CET       March 1, 2022 10:20:49 AM CET       Success
Provisioning Software Only GI with RHP   March 1, 2022 10:20:49 AM CET       March 1, 2022 10:20:49 AM CET       Success
Patch GI with RHP                        March 1, 2022 10:20:49 AM CET       March 1, 2022 10:30:06 AM CET       Success
Updating GIHome version                  March 1, 2022 10:30:06 AM CET       March 1, 2022 10:30:09 AM CET       Success
Validate GI availability                 March 1, 2022 10:30:12 AM CET       March 1, 2022 10:30:12 AM CET       Success
Patch KVM CRS type                       March 1, 2022 10:30:12 AM CET       March 1, 2022 10:30:13 AM CET       Success
Update System version                    March 1, 2022 10:30:13 AM CET       March 1, 2022 10:30:13 AM CET       Success
Cleanup JRE Home                         March 1, 2022 10:30:13 AM CET       March 1, 2022 10:30:13 AM CET       Success
Add SYSNAME in Env                       March 1, 2022 10:30:13 AM CET       March 1, 2022 10:30:13 AM CET       Success
Starting the clusterware                 March 1, 2022 10:30:13 AM CET       March 1, 2022 10:30:13 AM CET       Success
Setting ACL for disk groups              March 1, 2022 10:30:13 AM CET       March 1, 2022 10:30:17 AM CET       Success
Update lvm.conf file                     March 1, 2022 10:31:34 AM CET       March 1, 2022 10:31:34 AM CET       Success
Update previous workarounds              March 1, 2022 10:31:34 AM CET       March 1, 2022 10:31:34 AM CET       Success
preRebootNode Actions                    March 1, 2022 10:31:34 AM CET       March 1, 2022 10:36:07 AM CET       Success
Reboot Ilom                              March 1, 2022 10:36:07 AM CET       March 1, 2022 10:36:07 AM CET       Success

Server reboots 5 minutes after the patch ends. On my X8-2M this server patching lasted 25 minutes.

Let’s check the component’s versions now:

odacli describe-component | grep -v ^$ 
System Version
---------------
19.14.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.14.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.14.0.0.0           up-to-date
GI
                                          19.14.0.0.220118      up-to-date
DB
                                          19.13.0.0.211019      19.14.0.0.220118
DCSCONTROLLER
                                          19.14.0.0.0           up-to-date
DCSCLI
                                          19.14.0.0.0           up-to-date
DCSAGENT
                                          19.14.0.0.0           up-to-date
DCSADMIN
                                          19.14.0.0.0           up-to-date
OS
                                          7.9                   up-to-date
ILOM
                                          5.0.2.24.r141466      up-to-date
BIOS
                                          52050300              up-to-date
SHARED CONTROLLER FIRMWARE
                                          VDV1RL04              up-to-date
LOCAL DISK FIRMWARE
                                          1132                  up-to-date
SHARED DISK FIRMWARE
                                          1132                  up-to-date
HMP
                                          2.4.8.0.600           up-to-date

This looks fine.

Patching the storage

Patching the storage is only needed for older ODAs/patch levels. In case you need to apply a patch on the storage it’s easy: there is a pre-patch, and then the patch:

odacli create-prepatchreport -st -v 19.14.0.0.0
odacli update-storage -v 19.14.0.0.0

For HA ODAs using RAC, patching can be done in a rolling fashion:

odacli update-storage -v 19.14.0.0.0 --rolling

I never encountered troubles during storage patching, so it should be fine.

Patching the DB homes

Since 19.11, DB homes are created on an acfs filesystem. If you come from 19.10, you will need to configure this filesystem:

odacli configure-dbhome-storage -dg DATA

Time for patching the DB homes depends on the number of DB homes and number of databases. In this example, only one DB home is deployed:

odacli list-dbhomes

ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
5e7a146a-c18d-427b-a206-e532ec9907cf     OraDB19000_home1     19.13.0.0.211019                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 CONFIGURED

A prepatching is also needed here:

odacli create-prepatchreport -d -i 5e7a146a-c18d-427b-a206-e532ec9907cf -v 19.14.0.0.0
odacli describe-prepatchreport -i d7aa02c8-cd13-4729-891d-ea52fade3784

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  d7aa02c8-cd13-4729-891d-ea52fade3784
            Description:  Patch pre-checks for [DB, ORACHKDB]: DbHome is OraDB19000_home1
                 Status:  FAILED
                Created:  March 1, 2022 10:50:55 AM CET
                 Result:  One or more pre-checks failed for [ORACHK]

Node Name
---------------
dbi-oda-x8

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__DB__
Validate DB Home ID             Success   Validated DB Home ID:
                                          5e7a146a-c18d-427b-a206-e532ec9907cf
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate minimum agent version  Success   Validated minimum agent version
Is GI upgraded                  Success   Validated GI is upgraded
Validate available space for    Success   Validated free space required under
db                                        /u01/app/odaorahome
Validate dbHomesOnACFS          Success   User has configured diskgroup for
configured                                Database homes on ACFS
Validate Oracle base            Success   Successfully validated Oracle Base
Is DB clone available           Success   Successfully validated clone file
                                          exists
Evaluate DBHome patching with   Success   Successfully validated updating
RHP                                       dbhome with RHP
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Failed    Orachk validation failed: .
Validate command execution      Success   Validated command execution
Verify the Fast Recovery Area   Failed    AHF-2929: FRA space management
(FRA) has reclaimable space               problem file types are present
                                          without an RMAN backup completion
                                          within the last 7 days

I don’t care about Orachk recommendations on my database as this is a test system. I will apply the patch on this DB home:

odacli update-dbhome -i 5e7a146a-c18d-427b-a206-e532ec9907cf -v 19.14.0.0.0 -f
DCS-10802:Insufficient disk space on file system: /u01/app/odaorahome. Expected free space: 16GB, available space: 5GB

Default acfs filesystem size for db homes is quite small, and it does not extent automatically, so let’s increase its size:

odacli list-dbhome-storages


=============================================================================================================
ID                                     Node Description          Disk Group Volume      Size(GB)   Status
-------------------------------------- ---- -------------------- ---------- ----------- ---------- ----------
9f091fa4-a8de-4c02-9504-4c889c324598   0    ORACLE_HOME          DATA       orahome_sh  30         CONFIGURED
09747e05-d7d4-4f5d-94ab-b1eb7c730c74   0    ORACLE_BASE          DATA       odabase_n0  10         CONFIGURED
=============================================================================================================

odacli modify-dbhome-storage -i 9f091fa4-a8de-4c02-9504-4c889c324598 -s 60

odacli list-dbhome-storages


=============================================================================================================
ID                                     Node Description          Disk Group Volume      Size(GB)   Status
-------------------------------------- ---- -------------------- ---------- ----------- ---------- ----------
9f091fa4-a8de-4c02-9504-4c889c324598   0    ORACLE_HOME          DATA       orahome_sh  60         CONFIGURED
09747e05-d7d4-4f5d-94ab-b1eb7c730c74   0    ORACLE_BASE          DATA       odabase_n0  10         CONFIGURED
=============================================================================================================


odacli update-dbhome -i 5e7a146a-c18d-427b-a206-e532ec9907cf -v 19.14.0.0.0 -f

odacli describe-job -i 9293caea-6521-4e62-991d-cf66be1e10cf

Job details
----------------------------------------------------------------
                     ID:  9293caea-6521-4e62-991d-cf66be1e10cf
            Description:  DB Home Patching: Home Id is 5e7a146a-c18d-427b-a206-e532ec9907cf
                 Status:  Success
                Created:  March 1, 2022 11:09:06 AM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Adding USER SSH_EQUIVALENCE              March 1, 2022 11:09:12 AM CET       March 1, 2022 11:09:12 AM CET       Success
Adding USER SSH_EQUIVALENCE              March 1, 2022 11:09:12 AM CET       March 1, 2022 11:09:12 AM CET       Success
Adding USER SSH_EQUIVALENCE              March 1, 2022 11:09:12 AM CET       March 1, 2022 11:09:13 AM CET       Success
Creating wallet for DB Client            March 1, 2022 11:09:49 AM CET       March 1, 2022 11:09:49 AM CET       Success
Patch databases by RHP                   March 1, 2022 11:09:49 AM CET       March 1, 2022 11:13:06 AM CET       Success
updating database metadata               March 1, 2022 11:13:06 AM CET       March 1, 2022 11:13:06 AM CET       Success
Set log_archive_dest for Database        March 1, 2022 11:13:06 AM CET       March 1, 2022 11:13:09 AM CET       Success
Patch databases by RHP                   March 1, 2022 11:13:09 AM CET       March 1, 2022 11:17:19 AM CET       Success
updating database metadata               March 1, 2022 11:17:19 AM CET       March 1, 2022 11:17:19 AM CET       Success
Set log_archive_dest for Database        March 1, 2022 11:17:19 AM CET       March 1, 2022 11:17:22 AM CET       Success
Patch databases by RHP                   March 1, 2022 11:17:22 AM CET       March 1, 2022 11:22:08 AM CET       Success
updating database metadata               March 1, 2022 11:22:08 AM CET       March 1, 2022 11:22:08 AM CET       Success
Set log_archive_dest for Database        March 1, 2022 11:22:08 AM CET       March 1, 2022 11:22:10 AM CET       Success
Update System version                    March 1, 2022 11:22:10 AM CET       March 1, 2022 11:22:10 AM CET       Success
TDE parameter update                     March 1, 2022 11:23:02 AM CET       March 1, 2022 11:23:02 AM CET       Success

A new DB home has been created and my databases are now linked to this new one:

odacli list-dbhomes


ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
5e7a146a-c18d-427b-a206-e532ec9907cf     OraDB19000_home1     19.13.0.0.211019                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 CONFIGURED
530ce9aa-195c-4b45-98d4-5ab03855d47d     OraDB19000_home2     19.14.0.0.220118                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2 CONFIGURED

odacli list-databases

ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
42656d90-e18e-4325-a69a-13d3a6b58710     DBITST     SI       19.14.0.0.220118     true       OLTP     odb1     ASM        CONFIGURED   530ce9aa-195c-4b45-98d4-5ab03855d47d
a2193c80-d004-4295-847e-ec8277952bd7     DBTAG38    SI       19.14.0.0.220118     false      OLTP     odb1     ASM        CONFIGURED   530ce9aa-195c-4b45-98d4-5ab03855d47d
e7f9bf31-d185-4159-b579-181c75d6835b     CDBCBL     SI       19.14.0.0.220118     true       OLTP     odb1     ACFS       CONFIGURED   530ce9aa-195c-4b45-98d4-5ab03855d47d

The old DB home can now be removed safely:

odacli delete-dbhome -i 5e7a146a-c18d-427b-a206-e532ec9907cf

If your databases were created with 19.11 or earlier versions, a parameter needs to be changed:

su - oracle
. oraenv <<< DBITST
sqlplus / as sysdba
alter system set "_enable_numa_support"=true scope=spfile sid='*';
exit
srvctl stop database -d DBITST_SITE1
srvctl start database -d DBITST_SITE1

This only concerns multi-processor ODAs (not S ones) and it will force an instance to use local memory modules, those associated to the processor where the instance is running. This should improve overall performance.

Patching the other DB homes is done the same way.

Remember that patching standby databases may raise an error, as datapatch cannot be applied on a mounted or read only database.

I would recommand to check on each primary the patch level after patching each db home:

su – oracle
. oraenv <<< DBITST
sqlplus / as sysdba
set serverout on
exec dbms_qopatch.get_sqlpatch_status;
Patch Id : 33192694
	Action : APPLY
	Action Time : 10-NOV-2021 06:45:49
	Description : OJVM RELEASE UPDATE: 19.13.0.0.211019 (33192694)
	Logfile :
/u01/app/oracle/cfgtoollogs/sqlpatch/33192694/24421575/33192694_apply_G19794_CDB
ROOT_2021Nov10_06_29_39.log
	Status : SUCCESS

Patch Id : 33192793
	Action : APPLY
	Action Time : 10-NOV-2021 06:45:49
	Description : Database Release Update : 19.13.0.0.211019 (33192793)
	Logfile :
/u01/app/oracle/cfgtoollogs/sqlpatch/33192793/24462514/33192793_apply_G19794_CDB
ROOT_2021Nov10_06_29_39.log
	Status : SUCCESS

Patch Id : 33192694
	Action : ROLLBACK
	Action Time : 01-MAR-2022 11:17:15
	Description : OJVM RELEASE UPDATE: 19.13.0.0.211019 (33192694)
	Logfile :
/u01/app/odaorabase/oracle/cfgtoollogs/sqlpatch/33192694/24421575/33192694_rollb
ack_DBITST_CDBROOT_2022Mar01_11_15_15.log
	Status : SUCCESS

Patch Id : 33561310
	Action : APPLY
	Action Time : 01-MAR-2022 11:17:16
	Description : OJVM RELEASE UPDATE: 19.14.0.0.220118 (33561310)
	Logfile :
/u01/app/odaorabase/oracle/cfgtoollogs/sqlpatch/33561310/24538862/33561310_apply
_DBITST_CDBROOT_2022Mar01_11_15_43.log
	Status : SUCCESS

Patch Id : 33515361
	Action : APPLY
	Action Time : 01-MAR-2022 11:17:16
	Description : Database Release Update : 19.14.0.0.220118 (33515361)
	Logfile :
/u01/app/odaorabase/oracle/cfgtoollogs/sqlpatch/33515361/24589353/33515361_apply
_DBITST_CDBROOT_2022Mar01_11_15_43.log
	Status : SUCCESS

PL/SQL procedure successfully completed.
exit

Final checks

Let’s get the final versions:

odacli describe-component | grep -v ^$
System Version
---------------
19.14.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.14.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.14.0.0.0           up-to-date
GI
                                          19.14.0.0.220118      up-to-date
DB
                                          19.14.0.0.220118      up-to-date
DCSCONTROLLER
                                          19.14.0.0.0           up-to-date
DCSCLI
                                          19.14.0.0.0           up-to-date
DCSAGENT
                                          19.14.0.0.0           up-to-date
DCSADMIN
                                          19.14.0.0.0           up-to-date
OS
                                          7.9                   up-to-date
ILOM
                                          5.0.2.24.r141466      up-to-date
BIOS
                                          52050300              up-to-date
SHARED CONTROLLER FIRMWARE
                                          VDV1RL04              up-to-date
LOCAL DISK FIRMWARE
                                          1132                  up-to-date
SHARED DISK FIRMWARE
                                          1132                  up-to-date
HMP
                                          2.4.8.0.600           up-to-date

Everything is fine.

Cleanse the old patches

The old patches will never be used again, so don’t forget to remove previous patch files from the repository if your ODA has already been patched:

odacli cleanup-patchrepo -cl -comp db,gi -v 19.13.0.0.0

Put back your own settings

    Once everything is OK, don’t forget to put back your settings:

  • add your additional rpms manually if needed
  • put back your profile scripts for grid and oracle users
  • …

Patching an existing DB System

If you use DB Systems on your ODA, meaning that some of your databases are running in dedicated VMs, you will need to apply the patch inside each DB System.

odacli list-dbsystems
Name                  Shape       Cores  Memory      GI version          DB version          Status           Created                  Updated
--------------------  ----------  -----  ----------  ------------------  ------------------  ---------------  -----------------------  -----------------------
srvdb47               odb2        2      16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-22 17:40:42 CET  2022-02-22 18:13:44 CET
srvdb49               odb2        2      16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-22 16:27:54 CET  2022-02-22 16:58:43 CET
srvdb48               odb2        2      16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-22 15:53:25 CET  2022-02-22 16:23:22 CET

Let’s do a describe component in one of the DB systems:

ssh srvdb48
odacli describe-component | grep -v ^$
System Version
---------------
19.13.0.0.0
System node Name
---------------
srvdb48
Local System Version
---------------
19.13.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.13.0.0.0           19.14.0.0.0
GI                                        19.13.0.0.211019      19.14.0.0.220118
DB                                        19.13.0.0.211019      19.14.0.0.220118
DCSCONTROLLER                             19.13.0.0.0           19.14.0.0.0
DCSCLI                                    19.13.0.0.0           19.14.0.0.0
DCSAGENT                                  19.13.0.0.0           19.14.0.0.0
DCSADMIN                                  19.13.0.0.0           19.14.0.0.0
OS                                        7.9                   up-to-date

Patching method is the same as bare metal:

odacli update-dcsadmin -v 19.14.0.0.0
sleep 30; odacli update-dcscomponents -v 19.14.0.0.0
odacli update-dcsagent -v 19.14.0.0.0
sleep 180; odacli create-prepatchreport -s -v 19.14.0.0.0
odacli describe-prepatchreport -i 605cb646-9d39-4db4-bdcf-aca7f813942d


Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  605cb646-9d39-4db4-bdcf-aca7f813942d
            Description:  Patch pre-checks for [OS, GI, ORACHKSERVER]
                 Status:  FAILED
                Created:  March 1, 2022 11:48:18 AM CET
                 Result:  One or more pre-checks failed for [GI, ORACHK]

Node Name
---------------
srvdb48

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate BM versions            Failed
Validate kernel log level       Aborted   Internal error encountered
Validate minimum agent version  Aborted   Internal error encountered
Validate Central Inventory      Aborted   Internal error encountered
Validate patching locks         Aborted   Internal error encountered
Validate clones location exist  Aborted   Internal error encountered
Evaluate GI patching            Aborted   Internal error encountered
Validate command execution      Aborted   Internal error encountered

__ORACHK__
Running orachk                  Aborted   Internal error encountered
Validate command execution      Aborted   Internal error encountered

There is a bug with the repository mounted from the bare metal server, as described in the documentation:

cp /opt/oracle/oak/pkgrepos/System/VERSION /opt/oracle/oak/conf/VERSION
umount /opt/oracle/oak/pkgrepos
mount 192.168.17.2:/opt/oracle/oak/pkgrepos /opt/oracle/oak/pkgrepos

odacli create-prepatchreport -s -v 19.14.0.0.0
odacli describe-prepatchreport -i cfb24814-5c77-4ccb-9cee-eda971467a57

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  cfb24814-5c77-4ccb-9cee-eda971467a57
            Description:  Patch pre-checks for [OS, GI, ORACHKSERVER]
                 Status:  SUCCESS
                Created:  March 1, 2022 3:21:59 PM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
srvdb48

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate BM versions            Success   Validated BM server components
                                          versions
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

odacli update-server -v 19.14.0.0.0
odacli create-prepatchreport -d -i d8e0d75d-a810-4b89-805e-a4de5d646ec4 -v 19.14.0.0.0
odacli describe-prepatchreport -i 15065354-6a5d-411e-ad69-5991fc560428
odacli update-dbhome -i d8e0d75d-a810-4b89-805e-a4de5d646ec4 -v 19.14.0.0.0
odacli delete-dbhome -i d8e0d75d-a810-4b89-805e-a4de5d646ec4
odacli describe-component | grep -v ^$
System Version
---------------
19.14.0.0.0
System node Name
---------------
srvdb48
Local System Version
---------------
19.14.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.14.0.0.0           up-to-date
GI
                                          19.14.0.0.220118      up-to-date
DB
                                          19.14.0.0.220118      up-to-date
DCSCONTROLLER
                                          19.14.0.0.0           up-to-date
DCSCLI
                                          19.14.0.0.0           up-to-date
DCSAGENT
                                          19.14.0.0.0           up-to-date
DCSADMIN
                                          19.14.0.0.0           up-to-date
OS
                                          7.9                   up-to-date

Remember that using multiple DB Systems is nice but it’s much more work when you need to patch.

Conclusion

This release doesn’t bring any new feature and thus is easy to apply if you come from 19.10 or later versions. So don’t hesitate because this patch packs the latest bug and security fixes.

L’article Patch 19.14 is out for ODA: new features and how to apply est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/patch-19-14-is-out-for-oda-new-features-and-how-to-apply/feed/ 0 ODA 19.13: odacli update-dbhome raises DCS-10001 and PRGH-1153 https://www.dbi-services.com/blog/oda-19-13-odacli-update-dbhome-raises-dcs-10001-and-prgh-1153/ https://www.dbi-services.com/blog/oda-19-13-odacli-update-dbhome-raises-dcs-10001-and-prgh-1153/#respond Mon, 28 Feb 2022 14:13:23 +0000 https://www.dbi-services.com/blog/oda-19-13-odacli-update-dbhome-raises-dcs-10001-and-prgh-1153/ Introduction While patching ODAs from 19.10 to 19.13, the update-dbhome could fail if some databases inside this dbhome are stopped or only mounted. Job would then stop with a DCS-10001:Internal error encountered: PRGH-1153 : RHPHelper call to get runing nodes failed for DB: “ARITSTDB”. How to solve this problem? Environment description My environment is composed […]

L’article ODA 19.13: odacli update-dbhome raises DCS-10001 and PRGH-1153 est apparu en premier sur dbi Blog.

]]> Introduction

While patching ODAs from 19.10 to 19.13, the update-dbhome could fail if some databases inside this dbhome are stopped or only mounted. Job would then stop with a DCS-10001:Internal error encountered: PRGH-1153 : RHPHelper call to get runing nodes failed for DB: “ARITSTDB”.

How to solve this problem?

Environment description

My environment is composed of 3 X8-2M ODAs, 2 are dedicated for production with Data Guard, and 1 for test and development databases. Initialy deployed with 19.6, the goal was to patch to the latest 19.13. An intermediate patch was needed, so these ODAs will first be patched to 19.10 before applying 19.13 straight after.

Patching from 19.6 to 19.10 was fine, but patching the only 19c dbhome to 19.13 was not. Job raised an error:

odacli describe-job -i 087d618c-714c-463f-a1e8-7efcba0541fc

Job details
----------------------------------------------------------------
                     ID:  087d618c-714c-463f-a1e8-7efcba0541fc
            Description:  DB Home Patching: Home Id is 38c08a67-2730-4b76-92aa-4a47840082c5
                 Status:  Failure
                Created:  February 23, 2022 6:02:42 PM CET
                Message:  DCS-10001:Internal error encountered: PRGH-1153 : RHPHelper call to get runing nodes failed for DB: "ARITSTDB"..

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
DB Home Patching                         February 23, 2022 6:03:02 PM CET    February 23, 2022 7:02:53 PM CET    Failure
DB Home Patching                         February 23, 2022 6:03:03 PM CET    February 23, 2022 7:02:53 PM CET    Failure
Adding USER SSH_EQUIVALENCE              February 23, 2022 6:03:03 PM CET    February 23, 2022 6:03:03 PM CET    Success
Adding USER SSH_EQUIVALENCE              February 23, 2022 6:03:03 PM CET    February 23, 2022 6:03:04 PM CET    Success
task:TaskSequential_2030                 February 23, 2022 6:03:04 PM CET    February 23, 2022 7:01:55 PM CET    Failure
Creating wallet for DB Client            February 23, 2022 6:03:43 PM CET    February 23, 2022 6:03:43 PM CET    Success
Patch databases by RHP                   February 23, 2022 6:03:43 PM CET    February 23, 2022 6:11:03 PM CET    Success
updating database metadata               February 23, 2022 6:11:56 PM CET    February 23, 2022 6:11:56 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 6:11:56 PM CET    February 23, 2022 6:12:01 PM CET    Success
Patch databases by RHP                   February 23, 2022 6:12:01 PM CET    February 23, 2022 6:19:10 PM CET    Success
updating database metadata               February 23, 2022 6:20:03 PM CET    February 23, 2022 6:20:03 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 6:20:03 PM CET    February 23, 2022 6:20:08 PM CET    Success
Patch databases by RHP                   February 23, 2022 6:20:08 PM CET    February 23, 2022 6:25:26 PM CET    Success
updating database metadata               February 23, 2022 6:26:21 PM CET    February 23, 2022 6:26:21 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 6:26:21 PM CET    February 23, 2022 6:26:26 PM CET    Success
Patch databases by RHP                   February 23, 2022 6:26:26 PM CET    February 23, 2022 6:31:54 PM CET    Success
updating database metadata               February 23, 2022 6:32:49 PM CET    February 23, 2022 6:32:49 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 6:32:49 PM CET    February 23, 2022 6:32:53 PM CET    Success
Patch databases by RHP                   February 23, 2022 6:32:53 PM CET    February 23, 2022 6:38:09 PM CET    Success
updating database metadata               February 23, 2022 6:39:02 PM CET    February 23, 2022 6:39:03 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 6:39:03 PM CET    February 23, 2022 6:39:07 PM CET    Success
Patch databases by RHP                   February 23, 2022 6:39:07 PM CET    February 23, 2022 6:44:02 PM CET    Success
updating database metadata               February 23, 2022 6:44:58 PM CET    February 23, 2022 6:44:58 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 6:44:58 PM CET    February 23, 2022 6:45:03 PM CET    Success
Patch databases by RHP                   February 23, 2022 6:45:03 PM CET    February 23, 2022 6:51:55 PM CET    Success
updating database metadata               February 23, 2022 6:52:48 PM CET    February 23, 2022 6:52:48 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 6:52:48 PM CET    February 23, 2022 6:52:52 PM CET    Success
Patch databases by RHP                   February 23, 2022 6:52:52 PM CET    February 23, 2022 7:00:09 PM CET    Success
updating database metadata               February 23, 2022 7:01:05 PM CET    February 23, 2022 7:01:05 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 7:01:05 PM CET    February 23, 2022 7:01:10 PM CET    Success
Patch databases by RHP                   February 23, 2022 7:01:10 PM CET    February 23, 2022 7:01:55 PM CET    Failure

Status of databases after patching

Failure didn’t happen immediately, part of the job should have been done. Let’s check databases and version associated:

odacli list-databases

ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
6e26f06c-ee16-46f2-986e-0d82602fe93a     GOTTSTDB   SI       19.13.0.0.211019     false      OLTP     odb1     ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
17c84065-9bb9-4548-bf72-d0a02ad2213f     PATSTDB    SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
f34eddac-7794-47cd-9d6a-08aa3b2f90b7     PATSITDB   SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
185bc4bf-492e-4b98-9440-b24c2864f572     PATCIDB    SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
d0657336-b416-40d7-929f-8a9fe604e908     M4CTSTDB   SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
9681f86e-88b9-468f-93c6-9b2db76bede1     MINDEVDB   SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
a5e4f009-942a-488c-9097-6e34e6b858dd     MINTSTDB   SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
62416912-551b-44cb-a23d-8f60ca7d8a37     REGNTTST   SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
af37407a-f340-43b2-bb41-7f7e1143c703     LOGMTST    SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
6ce0e7a6-7560-4b2a-b992-cf6cca7500f0     WNTVSTDB   SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
9bebb050-8090-444f-bbc7-7dcfb4bc68a1     BQ3        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
d7da5dba-03d5-4cbd-ba7f-00aaea97c55c     POQ        SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
7e396dee-e4b3-4d84-8a4e-e42b4de69884     OQ1        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
64c248c3-aaf9-4a89-a672-f6e60e20e2aa     SQ1        SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
69b805d8-db8c-4240-aadc-e267b5c6f020     SQ2        SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
76562430-5554-4876-967f-58bcb81e2818     OD1        SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
4c50b2d6-0e4f-47d6-a278-5ee6bd4635d4     BD3        SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
9c5bac01-e9a6-4345-8076-4b647fb0457e     SD1        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
b051e5be-92c1-4d85-9060-7a36bf4befd6     POD        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
efd34b41-cbda-4785-9cda-e08450203619     SD2        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
4c679720-c577-4934-ab56-67d421538366     ARITSTDB   SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5

A few databases have been migrated to the new 19.13 dhbome, then job stopped after ARITSTDB database. Actually, ARITSTDB is a decommissioned database, still existing on my ODA. On one of the other ODA, behaviour was the same, but for another reason: databases were mostly standbys. So this part of the patch is not continuing after trying on a database not in read/write mode.

Retry to apply the patch on the dbhome

Let’s retry the patch to see if it can continue:

odacli update-dbhome -i 38c08a67-2730-4b76-92aa-4a47840082c5 -v 19.13.0.0.0 -f

odacli describe-job -i 9ec82ccd-d726-4172-be80-943bafbfbd18

Job details
----------------------------------------------------------------
                     ID:  9ec82ccd-d726-4172-be80-943bafbfbd18
            Description:  DB Home Patching: Home Id is 38c08a67-2730-4b76-92aa-4a47840082c5
                 Status:  Failure
                Created:  February 23, 2022 7:11:36 PM CET
                Message:  DCS-10001:Internal error encountered: PRCT-1014 : Internal error: CmdToolUtil-execute1-error1.

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
DB Home Patching                         February 23, 2022 7:11:50 PM CET    February 23, 2022 7:13:29 PM CET    Failure
DB Home Patching                         February 23, 2022 7:11:50 PM CET    February 23, 2022 7:13:29 PM CET    Failure
Adding USER SSH_EQUIVALENCE              February 23, 2022 7:11:50 PM CET    February 23, 2022 7:11:51 PM CET    Success
Adding USER SSH_EQUIVALENCE              February 23, 2022 7:11:51 PM CET    February 23, 2022 7:11:51 PM CET    Success
task:TaskSequential_3109                 February 23, 2022 7:11:51 PM CET    February 23, 2022 7:12:54 PM CET    Failure
Creating wallet for DB Client            February 23, 2022 7:12:30 PM CET    February 23, 2022 7:12:30 PM CET    Success
Patch databases by RHP                   February 23, 2022 7:12:30 PM CET    February 23, 2022 7:12:54 PM CET    Failure

Despite the failure status, my stopped database is now switched to its new home, but the task stopped again after this change.

odacli list-databases

ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
6e26f06c-ee16-46f2-986e-0d82602fe93a     GOTTSTDB   SI       19.13.0.0.211019     false      OLTP     odb1     ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
17c84065-9bb9-4548-bf72-d0a02ad2213f     PATSTDB    SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
f34eddac-7794-47cd-9d6a-08aa3b2f90b7     PATSITDB   SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
185bc4bf-492e-4b98-9440-b24c2864f572     PATCIDB    SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
d0657336-b416-40d7-929f-8a9fe604e908     M4CTSTDB   SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
9681f86e-88b9-468f-93c6-9b2db76bede1     MINDEVDB   SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
a5e4f009-942a-488c-9097-6e34e6b858dd     MINTSTDB   SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
62416912-551b-44cb-a23d-8f60ca7d8a37     REGNTTST   SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
af37407a-f340-43b2-bb41-7f7e1143c703     LOGMTST    SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
6ce0e7a6-7560-4b2a-b992-cf6cca7500f0     WNTVSTDB   SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
9bebb050-8090-444f-bbc7-7dcfb4bc68a1     BQ3        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
d7da5dba-03d5-4cbd-ba7f-00aaea97c55c     POQ        SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
7e396dee-e4b3-4d84-8a4e-e42b4de69884     OQ1        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
64c248c3-aaf9-4a89-a672-f6e60e20e2aa     SQ1        SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
69b805d8-db8c-4240-aadc-e267b5c6f020     SQ2        SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
76562430-5554-4876-967f-58bcb81e2818     OD1        SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
4c50b2d6-0e4f-47d6-a278-5ee6bd4635d4     BD3        SI       19.10.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   38c08a67-2730-4b76-92aa-4a47840082c5
9c5bac01-e9a6-4345-8076-4b647fb0457e     SD1        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
b051e5be-92c1-4d85-9060-7a36bf4befd6     POD        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
efd34b41-cbda-4785-9cda-e08450203619     SD2        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
4c679720-c577-4934-ab56-67d421538366     ARITSTDB   SI       19.13.0.0.210119     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9

Let’s start again the patching of this dbhome:

odacli update-dbhome -i 38c08a67-2730-4b76-92aa-4a47840082c5 -v 19.13.0.0.0 -f

odacli describe-job -i "36843376-2ea3-4d17-b641-dac34270d1f2"


Job details
----------------------------------------------------------------
                     ID:  36843376-2ea3-4d17-b641-dac34270d1f2
            Description:  DB Home Patching: Home Id is 38c08a67-2730-4b76-92aa-4a47840082c5
                 Status:  Success
                Created:  February 23, 2022 7:16:42 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Adding USER SSH_EQUIVALENCE              February 23, 2022 7:16:56 PM CET    February 23, 2022 7:16:57 PM CET    Success
Adding USER SSH_EQUIVALENCE              February 23, 2022 7:16:57 PM CET    February 23, 2022 7:16:57 PM CET    Success
Creating wallet for DB Client            February 23, 2022 7:17:36 PM CET    February 23, 2022 7:17:36 PM CET    Success
Patch databases by RHP                   February 23, 2022 7:17:36 PM CET    February 23, 2022 7:23:05 PM CET    Success
updating database metadata               February 23, 2022 7:23:59 PM CET    February 23, 2022 7:23:59 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 7:23:59 PM CET    February 23, 2022 7:24:03 PM CET    Success
Patch databases by RHP                   February 23, 2022 7:24:03 PM CET    February 23, 2022 7:31:11 PM CET    Success
updating database metadata               February 23, 2022 7:32:09 PM CET    February 23, 2022 7:32:09 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 7:32:09 PM CET    February 23, 2022 7:32:13 PM CET    Success
Patch databases by RHP                   February 23, 2022 7:32:13 PM CET    February 23, 2022 7:39:15 PM CET    Success
updating database metadata               February 23, 2022 7:40:09 PM CET    February 23, 2022 7:40:09 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 7:40:09 PM CET    February 23, 2022 7:40:14 PM CET    Success
Patch databases by RHP                   February 23, 2022 7:40:14 PM CET    February 23, 2022 7:47:25 PM CET    Success
updating database metadata               February 23, 2022 7:48:26 PM CET    February 23, 2022 7:48:26 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 7:48:26 PM CET    February 23, 2022 7:48:29 PM CET    Success
Patch databases by RHP                   February 23, 2022 7:48:29 PM CET    February 23, 2022 7:55:39 PM CET    Success
updating database metadata               February 23, 2022 7:56:34 PM CET    February 23, 2022 7:56:34 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 7:56:34 PM CET    February 23, 2022 7:56:39 PM CET    Success
Patch databases by RHP                   February 23, 2022 7:56:39 PM CET    February 23, 2022 8:03:39 PM CET    Success
updating database metadata               February 23, 2022 8:04:33 PM CET    February 23, 2022 8:04:33 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 8:04:34 PM CET    February 23, 2022 8:04:37 PM CET    Success
Patch databases by RHP                   February 23, 2022 8:04:37 PM CET    February 23, 2022 8:11:27 PM CET    Success
updating database metadata               February 23, 2022 8:12:19 PM CET    February 23, 2022 8:12:19 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 8:12:19 PM CET    February 23, 2022 8:12:23 PM CET    Success
Patch databases by RHP                   February 23, 2022 8:12:23 PM CET    February 23, 2022 8:18:00 PM CET    Success
updating database metadata               February 23, 2022 8:18:54 PM CET    February 23, 2022 8:18:54 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 8:18:54 PM CET    February 23, 2022 8:18:58 PM CET    Success
Patch databases by RHP                   February 23, 2022 8:18:58 PM CET    February 23, 2022 8:24:02 PM CET    Success
updating database metadata               February 23, 2022 8:24:56 PM CET    February 23, 2022 8:24:56 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 8:24:56 PM CET    February 23, 2022 8:25:00 PM CET    Success
Patch databases by RHP                   February 23, 2022 8:25:00 PM CET    February 23, 2022 8:30:13 PM CET    Success
updating database metadata               February 23, 2022 8:31:06 PM CET    February 23, 2022 8:31:06 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 8:31:06 PM CET    February 23, 2022 8:31:13 PM CET    Success
Patch databases by RHP                   February 23, 2022 8:31:13 PM CET    February 23, 2022 8:36:19 PM CET    Success
updating database metadata               February 23, 2022 8:37:15 PM CET    February 23, 2022 8:37:15 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 8:37:15 PM CET    February 23, 2022 8:37:19 PM CET    Success
Patch databases by RHP                   February 23, 2022 8:37:19 PM CET    February 23, 2022 8:42:41 PM CET    Success
updating database metadata               February 23, 2022 8:43:36 PM CET    February 23, 2022 8:43:36 PM CET    Success
Set log_archive_dest for Database        February 23, 2022 8:43:36 PM CET    February 23, 2022 8:43:39 PM CET    Success
Update System version                    February 23, 2022 8:43:39 PM CET    February 23, 2022 8:43:39 PM CET    Success
TDE parameter update                     February 23, 2022 8:44:27 PM CET    February 23, 2022 8:44:27 PM CET    Success

odacli list-databases
ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
6e26f06c-ee16-46f2-986e-0d82602fe93a     GOTTSTDB   SI       19.13.0.0.211019     false      OLTP     odb1     ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
17c84065-9bb9-4548-bf72-d0a02ad2213f     PATSTDB    SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
f34eddac-7794-47cd-9d6a-08aa3b2f90b7     PATSITDB   SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
185bc4bf-492e-4b98-9440-b24c2864f572     PATCIDB    SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
d0657336-b416-40d7-929f-8a9fe604e908     M4CTSTDB   SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
9681f86e-88b9-468f-93c6-9b2db76bede1     MINDEVDB   SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
a5e4f009-942a-488c-9097-6e34e6b858dd     MINTSTDB   SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
62416912-551b-44cb-a23d-8f60ca7d8a37     REGNTTST   SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
af37407a-f340-43b2-bb41-7f7e1143c703     LOGMTST    SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
6ce0e7a6-7560-4b2a-b992-cf6cca7500f0     WNTVSTDB   SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
9bebb050-8090-444f-bbc7-7dcfb4bc68a1     BQ3        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
d7da5dba-03d5-4cbd-ba7f-00aaea97c55c     POQ        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
7e396dee-e4b3-4d84-8a4e-e42b4de69884     OQ1        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
64c248c3-aaf9-4a89-a672-f6e60e20e2aa     SQ1        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
69b805d8-db8c-4240-aadc-e267b5c6f020     SQ2        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
76562430-5554-4876-967f-58bcb81e2818     OD1        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
4c50b2d6-0e4f-47d6-a278-5ee6bd4635d4     BD3        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
9c5bac01-e9a6-4345-8076-4b647fb0457e     SD1        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
b051e5be-92c1-4d85-9060-7a36bf4befd6     POD        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
efd34b41-cbda-4785-9cda-e08450203619     SD2        SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9
4c679720-c577-4934-ab56-67d421538366     ARITSTDB   SI       19.13.0.0.211019     false      OLTP     odb1s    ASM        CONFIGURED   e087a1ea-bf81-44a0-8f04-4d154c1fecc9

odacli finally managed to patch everything, but applying multiple time the patch on this home was needed. Number of runs depends on the number of databases not being read/write opened. As long as you have databases still pointing to the old dbhome you need to apply the patch again.

Conclusion

The solution to solve this problem is to start the patching again on the same dbhome and it will continue. Keep in mind that this has to be done in a sequential mode.

Patching an ODA takes time, time to prepare, time to backup prior patching, time to apply the patch, time to do the troubleshooting. But it works and everything is up to date with a single patch.

L’article ODA 19.13: odacli update-dbhome raises DCS-10001 and PRGH-1153 est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/oda-19-13-odacli-update-dbhome-raises-dcs-10001-and-prgh-1153/feed/ 0 ODA: Migrating from bare metal to CPU pools https://www.dbi-services.com/blog/oda-migrating-from-bare-metal-to-cpu-pools/ https://www.dbi-services.com/blog/oda-migrating-from-bare-metal-to-cpu-pools/#respond Tue, 22 Feb 2022 17:51:52 +0000 https://www.dbi-services.com/blog/oda-migrating-from-bare-metal-to-cpu-pools/ Introduction Oracle Database Appliance now fully supports virtualization with KVM, and this could be interesting for some of us. If you’re using Enterprise Edition, you probably decreased the number of cores to comply with your licenses. Now you can increase your core configuration with more cores, some being dedicated to your databases, and some for […]

L’article ODA: Migrating from bare metal to CPU pools est apparu en premier sur dbi Blog.

]]> Introduction

Oracle Database Appliance now fully supports virtualization with KVM, and this could be interesting for some of us. If you’re using Enterprise Edition, you probably decreased the number of cores to comply with your licenses. Now you can increase your core configuration with more cores, some being dedicated to your databases, and some for user-managed VMs. Let’s find out how you can migrate from bare-metal core allocation to CPU pools and discover the possibilities of this feature.

Updated the 1st of July, 2022: it looks like Bare-Metal CPU pools ARE NOT considered as hard-partitioning with Enterprise Edition. It means that despite using them for your Bare-Metal databases, all the enabled cores on your ODA need to be licensed. Those used for Bare-Metal CPU pools but also those remaining on the system, even if they are dedicated to VMs, and never used for your databases. The only supported mode for hard-partitioning is databases in DB Systems only, or core reduction according to your license on your ODA if you need Bare-Metal databases.

3 kinds of CPU pool

CPU pools are of 3 types on ODA:

  • Bare-Metal: these are CPU pools you can associate with Bare-Metal databases (must be licensed)
  • DB Systems: these are CPU pools you can associate with your DB Systems (must be licensed)
  • user-managed VMs: these are CPU pools for your multi-purpose VMs that don’t require any Oracle Database license

You can define multiple CPU pools of each kind, or none on your ODA.

This CPU pool configuration comes on top of your CPU core configuration. Previously, the only way to match your license with your ODA was to decrease the number of cores available on your system with odacli update-cpucore. Now it is much more flexible.

Configuration without using CPU pools

Without using CPU pools, on Enterprise Edition, core reduction is needed to match your licenses. For example, for 2 EE licenses on my ODA X8-2M, I need to lower my cores available to 4 after deployment:

odacli describe-cpucore


Node  Cores  Modified                           Job Status
----- ------ ---------------------------------- ---------------
0     32     February 17, 2022 1:55:24 PM CET   CONFIGURED

lscpu | grep MHz
CPU MHz:               2799.817
CPU max MHz:           3900.0000
CPU min MHz:           1000.0000

odacli update-cpucore -c 4
Modifying the enabled number of CPU requires a reboot of all nodes in the ODA system. Are you sure you want to proceed with this operation? (Y/N): Y

After the reboot, check new core configuration and CPU speed:

odacli describe-cpucore


Node  Cores  Modified                           Job Status
----- ------ ---------------------------------- ---------------
0     4      February 18, 2022 4:05:44 PM CET   CONFIGURED

lscpu | grep MHz
CPU MHz:               3899.943
CPU max MHz:           3900.0000
CPU min MHz:           1000.0000

As I already told you in another blog post, reducing the cores is giving more speed thanks to Turbo Boost. With only a few cores enabled, you can expect full speed from your Xeons.

Increase core configuration

On my ODA configured with 4 cores, I will first increase the number of cores on the server. I want to keep only 4 cores for databases according to my license.

odacli update-cpucore -c 8
Modifying the enabled number of CPU requires a reboot of all nodes in the ODA system. Are you sure you want to proceed with this operation? (Y/N): Y

Let’s check configuration and CPU speed after the reboot:

odacli describe-cpucore


Node  Cores  Modified                           Job Status
----- ------ ---------------------------------- ---------------
0     8      February 18, 2022 4:25:11 PM CET   CONFIGURED

lscpu | grep MHz
CPU MHz:               3685.555
CPU max MHz:           3900.0000
CPU min MHz:           1000.0000

As expected, my cores are running a little bit slower. Note that you’re not supposed to decrease the core numbers. You are only allowed to increase them after initial configuration. And according to your license, this sounds obvious.

Configuring and playing with the CPU pools

Now let’s create a CPU pool for bare metal databases:

odacli create-cpupool -n cpupool4BM -c 4 -bm
odacli describe-job -i a818a597-a60a-4824-a8e3-85c53d3087e6 | grep Status:
                 Status:  Success

And let’s associate this CPU pool to all my bare metal databases:

odacli list-databases

ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
42656d90-e18e-4325-a69a-13d3a6b58710     DBITST     SI       19.13.0.0.211019     true       OLTP     odb1     ASM        CONFIGURED   5e7a146a-c18d-427b-a206-e532ec9907cf
a2193c80-d004-4295-847e-ec8277952bd7     DBTAG38    SI       19.13.0.0.211019     false      OLTP     odb1     ASM        CONFIGURED   5e7a146a-c18d-427b-a206-e532ec9907cf


odacli modify-database -i 42656d90-e18e-4325-a69a-13d3a6b58710 -cp cpupool4BM
DB will be restarted as part of CPU Pool operation. Do you want to continue [y/n]:y


odacli describe-job -i ef4c6aff-c0e2-4493-9500-c3069559157e | grep Status:
                 Status:  Success

odacli modify-database -i a2193c80-d004-4295-847e-ec8277952bd7 -cp cpupool4BM
DB will be restarted as part of CPU Pool operation. Do you want to continue [y/n]:y

odacli describe-job -i c5e73921-619f-48dc-92c4-80be97128a23 | grep Status:
                 Status:  Success

Let’s create a CPU pool for user-managed VMs:

odacli create-cpupool -n cpupool4VM -c 4 -vm

odacli list-cpupools
Name                  Type                Configured on              Cores  Associated resources            Created                  Updated
--------------------  ------------------  -------------------------  -----  ------------------------------  -----------------------  -----------------------
cpupool4VM            VM                  dbi-oda-x8                 4      NONE                            2022-02-18 16:49:53 CET  2022-02-18 16:49:53 CET
cpupool4BM            BM                  dbi-oda-x8                 4      DBITST, DBTAG38                 2022-02-18 16:39:39 CET  2022-02-18 16:46:32 CET

Let’s create several VMs in this CPU pool:

odacli create-vmstorage -n VMstore -s 100G
odacli create-vm -n srvapp01 -m 8G -src /opt/dbi/AlmaLinux-8.4-x86_64-dvd.iso -vc 2 -cp cpupool4VM -vn pubnet -vms VMstore -s 16G -g "vnc,listen=10.36.0.241"
odacli create-vm -n srvapp02 -m 8G -src /opt/dbi/AlmaLinux-8.4-x86_64-dvd.iso -vc 2 -cp cpupool4VM -vn pubnet -vms VMstore -s 16G -g "vnc,listen=10.36.0.241"
odacli create-vm -n srvapp03 -m 8G -src /opt/dbi/AlmaLinux-8.4-x86_64-dvd.iso -vc 2 -cp cpupool4VM -vn pubnet -vms VMstore -s 16G -g "vnc,listen=10.36.0.241"

odacli list-vms
Name                  VM Storage            Node             Current State    Target State     Created                  Updated
--------------------  --------------------  ---------------  ---------------  ---------------  -----------------------  -----------------------
srvapp02              VMstore               dbi-oda-x8       ONLINE           ONLINE           2022-02-18 16:54:15 CET  2022-02-18 16:54:15 CET
srvapp01              VMstore               dbi-oda-x8       ONLINE           ONLINE           2022-02-18 16:53:45 CET  2022-02-18 16:53:45 CET
srvapp03              VMstore               dbi-oda-x8       ONLINE           ONLINE           2022-02-18 16:54:32 CET  2022-02-18 16:54:32 CET

No problem to configure 6 vCPUs on this 4-core CPU pool.

Normally I shouldn’t have enough cores to create a pool for DB Systems, let’s try:

odacli create-cpupool -n cpupool4DBS -c 4 -dbs

odacli describe-job  -i c0a27a41-469b-4145-bd1d-2bd770aeb5a7

Job details
----------------------------------------------------------------
                     ID:  c0a27a41-469b-4145-bd1d-2bd770aeb5a7
            Description:  CPU Pool cpupool4DBS creation
                 Status:  Failure
                Created:  February 18, 2022 4:55:45 PM CET
                Message:  DCS-10001:Internal error encountered: Not enough physical CPUs available for CPU Pool 'cpupool4DBS' on node 'dbi-oda-x8'.

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validate CPU Pool doesn't exist          February 18, 2022 4:55:45 PM CET    February 18, 2022 4:55:45 PM CET    Success
Validate NUMA state                      February 18, 2022 4:55:45 PM CET    February 18, 2022 4:55:45 PM CET    Success
Create metadata                          February 18, 2022 4:55:45 PM CET    February 18, 2022 4:55:45 PM CET    InternalError

Correct. This is not possible.

As I don’t want to add more licenses, let’s decrease cores on bare metal CPU pool. I will share my 2 licenses between BM and DBS:

odacli modify-cpupool -n cpupool4BM -c 2
WARNING: Must restart the following Oracle Databases associated with CPU Pool 'cpupool4BM' for the update to take effect: DBITST,DBTAG38

odacli describe-job -i 259d7fe1-afba-450f-beca-5410a5993160 | grep Status:
                 Status:  Success


odacli create-cpupool -n cpupool4DBS -c 2 -dbs

odacli describe-job -i f595cae9-fc00-4055-89e2-218d12307b06 | grep Status:
                 Status:  Success

I can now create some DB Systems using this CPU pool:

grep pool /opt/dbi/create_dbsystem_srvdb4*.json
/opt/dbi/create_dbsystem_srvdb47.json:        "cpuPoolName": "cpupool4DBS",
/opt/dbi/create_dbsystem_srvdb48.json:        "cpuPoolName": "cpupool4DBS",
/opt/dbi/create_dbsystem_srvdb49.json:        "cpuPoolName": "cpupool4DBS",

odacli create-dbsystem -p /opt/dbi/create_dbsystem_srvdb47.json
odacli create-dbsystem -p /opt/dbi/create_dbsystem_srvdb48.json
odacli create-dbsystem -p /opt/dbi/create_dbsystem_srvdb49.json

odacli list-dbsystems

Name                  Shape	  Cores  Memory      GI version          DB version          Status           Created                  Updated
--------------------  ----------  -----  ----------  ------------------  ------------------  ---------------  -----------------------  -----------------------
srvdb47               odb2        2	 16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-22 17:40:42 CET  2022-02-22 18:13:44 CET
srvdb49               odb2        2	 16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-22 16:27:54 CET  2022-02-22 16:58:43 CET
srvdb48               odb2        2	 16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-22 15:53:25 CET  2022-02-22 16:23:22 CET

odacli list-cpupools

Name                  Type                Configured on              Cores  Associated resources            Created                  Updated
--------------------  ------------------  -------------------------  -----  ------------------------------  -----------------------  -----------------------
cpupool4VM            VM                  dbi-oda-x8                 4      srvapp02, srvapp01, srvapp03    2022-02-18 16:49:53 CET  2022-02-18 16:49:53 CET
cpupool4BM            BM                  dbi-oda-x8                 2      DBITST, DBTAG38                 2022-02-18 16:39:39 CET  2022-02-18 16:58:21 CET
cpupool4DBS           DB_SYSTEM_SHARED    dbi-oda-x8                 2      srvdb47, srvdb49, srvdb48       2022-02-18 16:59:34 CET  2022-02-18 16:59:34 CET

As you can see, there is no problem configuring multiple DB Systems on a 2-core CPU pool, as there is no problem configuring 10 bare metal databases on a 2-core CPU pool.

Another interesting thing is that you can define multiple CPU pools of each kind. For example, if you want to dedicate cores to specific bare metal databases, you just need to create a new bare metal pool (here I also need to decrease another pool to free up 2 cores – and it will cost me 1 more EE license):

odacli modify-cpupool -n cpupool4VM -c 2
odacli create-cpupool -n cpupool4BMPROD  -c 2 -bm
odacli modify-database -i a2193c80-d004-4295-847e-ec8277952bd7 -cp cpupool4BMPROD

odacli list-cpupools

Name                  Type                Configured on              Cores  Associated resources            Created                  Updated
--------------------  ------------------  -------------------------  -----  ------------------------------  -----------------------  -----------------------
cpupool4VM            VM                  dbi-oda-x8                 2      srvapp02, srvapp01, srvapp03    2022-02-18 16:49:53 CET  2022-02-22 18:38:39 CET
cpupool4BMPROD        BM                  dbi-oda-x8                 2      DBTAG38                         2022-02-22 18:39:20 CET  2022-02-22 18:41:07 CET
cpupool4BM            BM                  dbi-oda-x8                 2      DBITST                          2022-02-18 16:39:39 CET  2022-02-22 18:41:06 CET
cpupool4DBS           DB_SYSTEM_SHARED    dbi-oda-x8                 2      srvdb47, srvdb49, srvdb48       2022-02-18 16:59:34 CET  2022-02-18 16:59:34 CET

Conclusion

CPU pools are a brilliant addition to ODA. And it works like a charm. As this is still compatible with core reduction, you also benefit from high CPU core speed and load isolation. Bravo!

L’article ODA: Migrating from bare metal to CPU pools est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/oda-migrating-from-bare-metal-to-cpu-pools/feed/ 0 Oracle Database Appliance: Install a DB Release Update manually https://www.dbi-services.com/blog/oracle-database-appliance-install-a-db-release-update-manually/ https://www.dbi-services.com/blog/oracle-database-appliance-install-a-db-release-update-manually/#respond Tue, 15 Feb 2022 20:18:49 +0000 https://www.dbi-services.com/blog/oracle-database-appliance-install-a-db-release-update-manually/ By Clemens Bleile Recently I upgraded an ODA X7-M from 19.12. to 19.13. After the dcs-, server- and storage-upgrade several databases on the machine had to be patched from 19.9. to 19.13. The [root@<node> ~]# odacli create-prepatchreport --dbhome --dbhomeid <home-id> -v 19.13.0.0.0 went through without reporting an issue, but during the [root@<node> ~]# odacli update-dhome […]

L’article Oracle Database Appliance: Install a DB Release Update manually est apparu en premier sur dbi Blog.

]]> By Clemens Bleile

Recently I upgraded an ODA X7-M from 19.12. to 19.13. After the dcs-, server- and storage-upgrade several databases on the machine had to be patched from 19.9. to 19.13. The 

[root@<node> ~]# odacli create-prepatchreport --dbhome --dbhomeid <home-id> -v 19.13.0.0.0

went through without reporting an issue, but during the

[root@<node> ~]# odacli update-dhome -i <home-id> -v 19.13.0.0.0 -f

I got an error “DCS-10001:Internal error encountered: null.”:

[root@<node> ~]# odacli describe-job -i "<Job-Id>"

Job details
----------------------------------------------------------------
                     ID:  <Job-Id>
            Description:  DB Home Patching: Home Id is <Home-Id>
                 Status:  Failure
                Created:  February 13, 2022 1:28:41 PM CET
                Message:  DCS-10001:Internal error encountered: null.

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
DB Home Patching                         February 13, 2022 1:29:02 PM CET    February 13, 2022 1:33:28 PM CET    Failure
DB Home Patching                         February 13, 2022 1:29:02 PM CET    February 13, 2022 1:33:28 PM CET    Failure
Adding USER SSH_EQUIVALENCE              February 13, 2022 1:29:02 PM CET    February 13, 2022 1:29:02 PM CET    Success
Adding USER SSH_EQUIVALENCE              February 13, 2022 1:29:02 PM CET    February 13, 2022 1:29:03 PM CET    Success
Adding USER SSH_EQUIVALENCE              February 13, 2022 1:29:03 PM CET    February 13, 2022 1:29:04 PM CET    Success
task:TaskSequential_3705                 February 13, 2022 1:29:04 PM CET    February 13, 2022 1:33:04 PM CET    Failure
Creating wallet for DB Client            February 13, 2022 1:29:44 PM CET    February 13, 2022 1:29:44 PM CET    Success
Patch databases by RHP                   February 13, 2022 1:29:44 PM CET    February 13, 2022 1:31:28 PM CET    Success
updating database metadata               February 13, 2022 1:32:22 PM CET    February 13, 2022 1:32:22 PM CET    Success
Set log_archive_dest for Database        February 13, 2022 1:32:22 PM CET    February 13, 2022 1:32:25 PM CET    Success
Patch databases by RHP                   February 13, 2022 1:32:25 PM CET    February 13, 2022 1:33:04 PM CET    Failure

[root@<node> ~]#

Checking the RHP logfile /opt/oracle/rhp/rhplog/rhpapi.log.0 showed the following:

[UID:<UID>] [Patch databases by RHP : JobId=<Job-Id>] [ 2022-02-13 13:33:04.671 CET ] [BatchMoveOpImpl.internalContinueMove:3348]  Batch failed for at least one DB
[UID:<UID>] [Patch databases by RHP : JobId=lt.Job-Id>] [ 2022-02-13 13:33:04.671 CET ] [DBPatchUpgradeOperationImpl.move:477]  attempt to move or upgrade database failed with OperationException : PRCT-1003 : failed to run "rhphelper" on node "<node>"
PRCT-1014 : Internal error: RHPHELP112_mergeLsnr-08
[UID:<UID>] [Patch databases by RHP : JobId=lt.Job-Id>] [ 2022-02-13 13:33:04.671 CET ] [DatabaseOperationImpl.move:1448]  OperationException: PRCT-1003 : failed to run "rhphelper" on node "<node>"
PRCT-1014 : Internal error: RHPHELP112_mergeLsnr-08
[UID:<UID>] [Patch databases by RHP : JobId=lt.Job-Id>] [ 2022-02-13 13:33:04.671 CET ] [GHOperationCommonImpl.moveDatabase:2978]  OperationException: PRCT-1003 : failed to run "rhphelper" on node "<node>"
PRCT-1014 : Internal error: RHPHELP112_mergeLsnr-08
[UID:<UID>] [Patch databases by RHP : JobId=lt.Job-Id>] [ 2022-02-13 13:33:04.671 CET ] [FPPMBeanImpl.doOp:372]  InvocationTargetException caught
[UID:<UID>] [Patch databases by RHP : JobId=lt.Job-Id>] [ 2022-02-13 13:33:04.672 CET ] [FPPMBeanImpl.doOp:382]  Exception:
java.lang.NullPointerException: null

So basically the errors were

PRCT-1003 : failed to run "rhphelper" on node "<node>"
PRCT-1014 : Internal error: RHPHELP112_mergeLsnr-08

The analysis revealed the root cause for the issue to be Bug 32833813. This is described in My Oracle Support Note 32833813.8 for an Upgrade from 11.2. to 19.2., but it obviously may also happen when moving to a new ORACLE_HOME for an Release Update with RHP:

Upgrading 11.2 SI Database to 19c Failed With Error PRCT-1003 : failed to run “rhphelper” on node “<HOSTNAME>” PRCT-1014 : internal error: rhphelp12102_main-02

The bug is fixed in the 19.14.0.0.220118 (JAN 2022) OCW Release Update. According the MOS Note there is no workaround.

The workaround for me was to change the ORACLE_HOME in the cluster registry to 19.13. for all DBs and finally update the ODA registry. I.e. in my case with the 19.13.-ORACLE_HOME being /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_3:

[oracle@<node> ~]$ srvctl stop database -db <DBNAME>
[oracle@<node> ~]$ srvctl modify database -db <DBNAME> -oraclehome /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_3
[oracle@<node> ~]$ srvctl config database -db <DBNAME>
[oracle@<node> ~]$ srvctl start database -db <DBNAME>
[oracle@<node> ~]$ . oraenv
ORACLE_SID = [oracle] ? <ORACLE_SID>
[oracle@<node> ~]$ cd $ORACLE_HOME/OPatch
[oracle@<node> ~]$ ./datapatch -verbose

REMARKs:
– After modifying the ORACLE_HOME with srvctl the agent automatically updates the /etc/oratab.
– On a standby-DB it’s of course not necessary to run datapatch.

To automate this I wrote a bash-script, so that I can start this against all my databases on this server running in the 19.9.-ORACLE_HOME.

The last step is to update the ODA registry to reflect the change of the databases ORACLE_HOME:

[root@<node> ~]# odacli update-registry -n db -f

Summary: Sometimes it’s necessary to manually patch databases to a newer RU on the ODA without using odacli. Fortunately we have the command “odacli update-registry” to update the ODA registry concerning our manual change.

L’article Oracle Database Appliance: Install a DB Release Update manually est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/oracle-database-appliance-install-a-db-release-update-manually/feed/ 0 Oracle Database Appliance (ODA): Adjusting to the new ORACLE_BASE-setting from 19.11. onwards https://www.dbi-services.com/blog/oracle-database-appliance-oda-adjusting-to-the-new-oracle_base-setting-from-19-11-onwards/ https://www.dbi-services.com/blog/oracle-database-appliance-oda-adjusting-to-the-new-oracle_base-setting-from-19-11-onwards/#respond Fri, 21 Jan 2022 20:58:53 +0000 https://www.dbi-services.com/blog/oracle-database-appliance-oda-adjusting-to-the-new-oracle_base-setting-from-19-11-onwards/ By Clemens Bleile From Oracle Database Appliance (ODA) version 19.11. onwards the ORACLE_HOMEs and ORACLE_BASE will be created on ACFS, i.e. take space from an ASM-diskgroup instead of taking filesystem space from /u01. See this blog concerning details. If the ODA has been upgraded from a release < 19.11. to 19.11. or newer, then you […]

L’article Oracle Database Appliance (ODA): Adjusting to the new ORACLE_BASE-setting from 19.11. onwards est apparu en premier sur dbi Blog.

]]> By Clemens Bleile

From Oracle Database Appliance (ODA) version 19.11. onwards the ORACLE_HOMEs and ORACLE_BASE will be created on ACFS, i.e. take space from an ASM-diskgroup instead of taking filesystem space from /u01. See this blog concerning details.

If the ODA has been upgraded from a release < 19.11. to 19.11. or newer, then you usually have diag- and admin-data in 2 ORACLE_BASE-locations: 

/u01/app/oracle

for databases created before migrating to ODA 19.11. or newer and

/u01/app/odaorabase/oracle

for databases created after the migration to ODA 19.11. or newer.

An ORACLE_HOME is associated to an ORACLE_BASE. This can be shown by running the command orabase:

oracle@dbi-oda-x8:/home/oracle/ [rdbms191300] orabase
/u01/app/odaorabase/oracle

orabase takes its info from 

$ORACLE_HOME/install/orabasetab

Migrating databases created before ODA 19.11. to an ORACLE_HOME created after the installation of ODA 19.11. causes diagnostics and admin-data to remain in the “old” ORACLE_BASE /u01/app/oracle, but the command orabase points to the “new” ORACLE_BASE /u01/app/odaorabase/oracle. Tools, which rely on a global ORACLE_BASE settings may have problems handling 2 ORACLE_BASE locations on a machine.

REMARK: The dbi-services tool DMK Management Kit can handle the situation with different ORACLE_BASE on a machine by specifying the ORACLE_BASE-location per database in separate sections in $DMK_HOME/etc/dmk.conf.

The following instructions will show the steps to move the diag- and admin-data of a database from an “old” ORACLE_BASE /u01/app/oracle to the “new” ORACLE_BASE /u01/app/odaorabase/oracle:

E.g. assume we have a Database CBLTEST which has an ORACLE_BASE /u01/app/oracle and a newly created database CBLTEST2 with an ORACLE_BASE /u01/app/odaorabase0:

[root@dbi-oda-x8 log]# odacli list-databases
 
ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID                                
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
f9e51c56-fcd2-47a3-bcb4-56587a710f1e     CBLTEST    SI       19.9.0.0.201020      false      OLTP     odb1     ASM        CONFIGURED   d9d1b8dd-4abc-46fe-aae4-d52162d66dab    
34fd593b-7586-41c9-a58d-71dfc0cf1d90     CBLTEST2   SI       19.13.0.0.211019     true       OLTP     odb1s    ASM        CONFIGURED   4dda330d-feed-491b-bace-006c22b75672

After bringing CBLTEST to 19.13. with

# odacli update-dbhome -i d9d1b8dd-4abc-46fe-aae4-d52162d66dab -v 19.13.0.0.0

it has the following ORACLE_BASE-setting:

oracle@dbi-oda-x8:/home/oracle/ [CBLTEST] orabase
/u01/app/odaorabase/oracle

but the diagnostics-data still below the “old” ORACLE_BASE.

Here the steps to copy your diag- and admin-data for CBLTEST to the new ORACLE_BASE:

REMARK 1: The situation with Read-Only ORACLE_HOMEs is not covered here.
REMARK 2: Before doing this in production you should test this on a test system carefully. If in doubt open a Service Request with Oracle.

1. Change diagnostic_dest and the audit_file_dest in the spfile

sqlplus / as sysdba
SQL> alter system set diagnostic_dest='/u01/app/odaorabase/oracle' scope=spfile;
SQL> alter system set audit_file_dest='/u01/app/odaorabase/oracle/admin/CBLTEST/adump' scope=spfile;

2. Shutdown the DB

srvctl stop database -db CBLTEST

3. Copy the rdbms-diag-data to new ORACLE_BASE

cp -pR /u01/app/oracle/diag/rdbms/cbltest /u01/app/odaorabase/oracle/diag/rdbms

4. Copy the admin-directory to the new ORACLE_BASE

cp -pR /u01/app/oracle/admin/CBLTEST /u01/app/odaorabase/oracle/admin

5. Copy the audit-data for spillover-files from unified auditing. 

cp -pR /u01/app/oracle/audit/CBLTEST /u01/app/odaorabase/oracle/audit

6. Change ORACLE_BASE to the new value and adjust your scripts, which do set ORACLE_BASE. E.g. for DMK adjust $DMK_HOME/etc/dmk.conf

export ORACLE_BASE=/u01/app/odaorabase/oracle

REMARK: You may check if the environment variable ORACLE_BASE has been stored in the Oracle Cluster registry:

oracle@dbi-oda-x8:/home/oracle/ [CBLTEST] srvctl getenv database -d CBLTEST -t "ORACLE_BASE"
CBLTEST:
PRKF-1128 : Environment variable ORACLE_BASE is not defined.

If it has been set (which would be very unusuable on an ODA) then you can overwrite it with

oracle@dbi-oda-x8:/home/oracle/ [CBLTEST] srvctl setenv database -d CBLTEST -t "ORACLE_BASE=/u01/app/odaorabase/oracle"
oracle@dbi-oda-x8:/home/oracle/ [CBLTEST] srvctl getenv database -d CBLTEST -t "ORACLE_BASE"
CBLTEST:
ORACLE_BASE=/u01/app/odaorabase/oracle

7. Startup the DB

srvctl start database -db CBLTEST

8. Check that everything is correct

8.1. Re-login as oracle, set the environment and connect “as sysdba”. Here with DMK sourced:

oracle@dbi-oda-x8:/home/oracle/ [rdbms191300] CBLTEST
********* dbi services Ltd. *********
STATUS                 : OPEN
DB_UNIQUE_NAME         : CBLTEST
OPEN_MODE              : READ WRITE
LOG_MODE               : ARCHIVELOG
DATABASE_ROLE          : PRIMARY
FLASHBACK_ON           : NO
FORCE_LOGGING          : YES
VERSION                : 19.13.0.0.0
CDB Enabled            : NO
*************************************
oracle@dbi-oda-x8:/home/oracle/ [CBLTEST] echo $ORACLE_BASE
/u01/app/odaorabase/oracle
oracle@dbi-oda-x8:/home/oracle/ [CBLTEST] sqh

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Jan 21 20:51:48 2022
Version 19.13.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.

Connected to:
Oracle Database 19c Standard Edition 2 Release 19.0.0.0.0 - Production
Version 19.13.0.0.0

SQL> select value from v$diag_info where value like '/u01%';

VALUE
---------------------------------------------------------------------------------
/u01/app/odaorabase/oracle
/u01/app/odaorabase/oracle/diag/rdbms/cbltest/CBLTEST
/u01/app/odaorabase/oracle/diag/rdbms/cbltest/CBLTEST/trace
/u01/app/odaorabase/oracle/diag/rdbms/cbltest/CBLTEST/alert
/u01/app/odaorabase/oracle/diag/rdbms/cbltest/CBLTEST/incident
/u01/app/odaorabase/oracle/diag/rdbms/cbltest/CBLTEST/cdump
/u01/app/odaorabase/oracle/diag/rdbms/cbltest/CBLTEST/hm
/u01/app/odaorabase/oracle/diag/rdbms/cbltest/CBLTEST/trace/CBLTEST_ora_45735.trc
/u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4

9 rows selected.

SQL> quit
Disconnected from Oracle Database 19c Standard Edition 2 Release 19.0.0.0.0 - Production
Version 19.13.0.0.0
oracle@dbi-oda-x8:/home/oracle/ [CBLTEST] cda
oracle@dbi-oda-x8:/u01/app/odaorabase/oracle/admin/CBLTEST/ [CBLTEST] ls -l
total 364
drwxr-x--- 2 oracle oinstall 20480 Jan 13 22:01 adump
drwxr-xr-x 2 oracle oinstall 20480 Jan 13 22:48 backup
drwxr-x--- 2 oracle oinstall 20480 Jan 13 12:27 dpdump
drwxr-xr-x 2 oracle oinstall 20480 Jan 13 18:01 etc
drwxr-xr-x 2 oracle oinstall 20480 Jan 13 22:48 log
drwxr-x--- 2 oracle oinstall 20480 Jan 13 12:29 pfile
drwxr-x--- 2 oracle oinstall 20480 Jan 13 12:23 xdb_wallet
oracle@dbi-oda-x8:/u01/app/odaorabase/oracle/admin/CBLTEST/ [CBLTEST]

8.2. Check adrci

oracle@dbi-oda-x8:/home/oracle/ [CBLTEST] adrci

ADRCI: Release 19.0.0.0.0 - Production on Fri Jan 21 19:51:30 2022

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

ADR base = "/u01/app/odaorabase/oracle"
adrci> show base
ADR base is "/u01/app/odaorabase/oracle"
adrci>

–> The Base-setting in adrci is coming from the file $ORACLE_HOME/log/diag/adrci_dir.mif

9. Cleanup

If everything is correct, then you may delete the data from the “old” ORACLE_BASE:

rm -rf /u01/app/oracle/diag/rdbms/cbltest
rm -rf /u01/app/oracle/admin/CBLTEST
rm -rf /u01/app/oracle/admin/audit/CBLTEST

References

https://marcelpils.de/orabase-doesnt-show-the-current-oracle-base-path
https://www.thegeekdiary.com/oracle-database-how-to-set-environment-variables-using-srvctl
https://logic.edchen.org/how-to-resolve-no-adr-base-is-set

MOS Notes

How To Change The Value For ORACLE_BASE In The Inventory Of A 12.1 RDBMS Home (Doc ID 2010941.1)
orabase command returns no value instead of ORACLE_BASE value (Doc ID 2225573.1)

L’article Oracle Database Appliance (ODA): Adjusting to the new ORACLE_BASE-setting from 19.11. onwards est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/oracle-database-appliance-oda-adjusting-to-the-new-oracle_base-setting-from-19-11-onwards/feed/ 0 Oracle Database Appliance 19.13: what’s new and how to patch? https://www.dbi-services.com/blog/oracle-database-appliance-19-13-whats-new-and-how-to-patch/ https://www.dbi-services.com/blog/oracle-database-appliance-19-13-whats-new-and-how-to-patch/#respond Sat, 11 Dec 2021 09:55:01 +0000 https://www.dbi-services.com/blog/oracle-database-appliance-19-13-whats-new-and-how-to-patch/ Introduction Patch 19.13 is now available on ODA. It’s time to test it. What’s new? This version brings October’s PSU to database and grid homes with their bug fixes, as usual. It also brings 21.4 for DB Systems (21c being an innovation release, understand just for testing). Something new that some of us have been […]

L’article Oracle Database Appliance 19.13: what’s new and how to patch? est apparu en premier sur dbi Blog.

]]> Introduction

Patch 19.13 is now available on ODA. It’s time to test it.

What’s new?

This version brings October’s PSU to database and grid homes with their bug fixes, as usual. It also brings 21.4 for DB Systems (21c being an innovation release, understand just for testing). Something new that some of us have been waiting for a while, a multi-user mode for odacli tasks: you don’t need to be root anymore for managing the appliance. This new feature can only be enabled when provisioning, so don’t expect to catch it after patching. A parameter deployment feature is also available now, helping managing parameters across multiple instances (bare metal only).

This 19.13 will be the latest one for those still using virtualized ODA: virtualization is now integrated in all bare metal ODAs with KVM (instead of OVM), check this blog post if you’re not aware about that. Plan to redeploy your virtualized ODA to bare metal if your system is still supported for a couple of years.

Which ODA is compatible with 19.13?

For sure, ODA X8 in its 3 flavors (S, M, HA) is compatible as this is the current one (but it should be replaced quite soon). X7 and X6 are also compatible without any problem. X5-2HA is also still supported, this could be one of the latest patch for this nearly 7-year old ODA.

Is this patch a cumulative one?

Most of the patches are cumulative, meaning that you can apply them on top of a version older than the previous one. This 19.13 can be applied on top of 19.9 or later, 19.9 being 1 year old. This is something I would recommend to each of my customer: apply patches at least once a year to avoid longer patching, longer meaning multiple patches to apply to reach target version.

If you are still using an old version, I would say 18.x or older, consider reimaging instead of patching. Reimaging is much more comfortable and cleaner compared to 3 or more patches to apply.

Is there also a patch for my databases?

If you’re using 12.1, 12.2 or 19c, these databases can be patched. 11gR2 and 18c are no more supported on ODA, meaning that your databases will continue working but you should definitely consider an upgrade quite soon. Remember that if you choose to reimage to this latest version, you will not find any 18c or 11gR2 DB homes to deploy.

Download the patch and clone files

Since months now, the patch is lighter because it doesn’t include DB and GI patches anymore. Patching DB or GI is deploying a new home and migrating from the older one. As this new way of patching is based on DB and GI clones, download the corresponding clones to be able to apply the complete patch.

33518928 => the patch itself
30403673 => the GI clone needed for deploying new version (mandatory)
30403662 => the DB clone for deploying new version of 19c (if you use 19c databases)
27119402 => the DB clone for deploying new version of 12.2 (if you use 12.2 databases)
23494992 => the DB clone for deploying new version of 12.1 (if you use 12.1 databases)

Be sure to choose the very latest 19.13 when downloading, as most of the time My Oracle Support will propose an older version for GI clones and DB clones (patch number is the same for older versions).

Prepare the patching

A pre-patch is now provided and needed before applying an ODA patch, but prior running it, please check these prerequisites:

  • filesystems have 20% available free space (does not concern acfs volumes)
  • additional rpms manually installed should be removed
  • revert profile scripts to default’s one (concerns grid and oracle users)
  • make sure you can afford longer than planned downtime, 4 hours being the bare minimum for patching and… troubleshooting. 1 day is never too much.

Version precheck

Start to check current version on all components:

odacli describe-component | grep -v ^$
System Version
---------------
19.12.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.12.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.12.0.0.0           up-to-date
GI                                        19.12.0.0.210720      up-to-date
DB                                        19.12.0.0.210720      up-to-date
DCSCONTROLLER                             19.12.0.0.0           up-to-date
DCSCLI                                    19.12.0.0.0           up-to-date
DCSAGENT                                  19.12.0.0.0           up-to-date
DCSADMIN                                  19.12.0.0.0           up-to-date
OS                                        7.9                   up-to-date
ILOM                                      5.0.2.24.r141466      up-to-date
BIOS                                      52050300              up-to-date
FIRMWARECONTROLLER                        VDV1RL04              up-to-date
FIRMWAREDISK                              1132                  up-to-date
HMP                                       2.4.8.0.600           up-to-date

Once the patch will be registered in the ODA repository, the “Available Version” column will be fed with versions provided within the patch.

On my ODA X8-2M, I only need 19c databases. As I’m already running on previous 19.12 version, patching shouldn’t be too risky.

Prepare the files and update the DCS tools

Copy the patch files to your ODA in a temp directory. Then unzip the files:

cd /opt/dbi/
for f in p*1913000*.zip; do unzip -n $f; done
Archive:  p30403662_1913000_Linux-x86-64.zip
 extracting: odacli-dcs-19.13.0.0.0-211109-DB-19.13.0.0.zip
Archive:  p30403673_1913000_Linux-x86-64.zip
 extracting: odacli-dcs-19.13.0.0.0-211109-GI-19.13.0.0.zip
Archive:  p33518928_1913000_Linux-x86-64.zip
 extracting: oda-sm-19.13.0.0.0-211127-server.zip

rm -rf p*1913000*.zip

Register the patch in the repository:

odacli update-repository -f /opt/dbi/oda-sm-19.13.0.0.0-211127-server.zip

odacli describe-component | grep -v ^$
System Version
---------------
19.12.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.12.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.12.0.0.0           19.13.0.0.0
GI                                        19.12.0.0.210720      19.13.0.0.211019
DB                                        19.12.0.0.210720      19.13.0.0.211019
DCSCONTROLLER                             19.12.0.0.0           19.13.0.0.0
DCSCLI                                    19.12.0.0.0           19.13.0.0.0
DCSAGENT                                  19.12.0.0.0           19.13.0.0.0
DCSADMIN                                  19.12.0.0.0           19.13.0.0.0
OS                                        7.9                   up-to-date
ILOM                                      5.0.2.24.r141466      up-to-date
BIOS                                      52050300              up-to-date
FIRMWARECONTROLLER                        VDV1RL04              up-to-date
FIRMWAREDISK                              1132                  up-to-date
HMP                                       2.4.8.0.600           up-to-date

Don’t update the repository with GI and DB clones now because you will receive this kind of error:

DCS-10001:Internal error encountered: Cannot find the corresponding image for /opt/dbi/odacli-dcs-19.13.0.0.0-211109-GI-19.13.0.0.zip in img_metadata.

Update the DCS tooling of your ODA:

odacli update-dcsadmin -v 19.13.0.0.0
odacli update-dcscomponents -v 19.13.0.0.0
odacli update-dcsagent -v 19.13.0.0.0

Note that updating the DCS components is not done through a job:

odacli list-jobs | head -n 3;  odacli list-jobs | tail -n 4
ID                                       Description                      Created                             Status
---------------------------------------- -------------------------------- ----------------------------------- ---------
9eccda54-418f-4ac8-8196-197a17fe1f48     Repository Update                                                           December 9, 2021 11:01:49 AM CET    Success
d1daaedd-f522-448e-8f67-08db0e80b6db     DcsAdmin patching                                                           December 9, 2021 3:23:45 PM CET     Success
227f0207-85d0-4d4c-bee8-61545241a588     DcsAgent patching                                                           December 9, 2021 3:26:15 PM CET     Success

Now you can register GI and DB clones:

odacli update-repository -f /opt/dbi/odacli-dcs-19.13.0.0.0-211109-GI-19.13.0.0.zip
odacli update-repository -f /opt/dbi/odacli-dcs-19.13.0.0.0-211109-DB-19.13.0.0.zip

odacli list-jobs | head -n 3;  odacli list-jobs | tail -n 3
ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
53e78d25-f594-4784-a320-aa3df5e996a6     Repository Update                                                           December 9, 2021 3:48:14 PM CET     Success
38c4202e-b1e3-4ba7-8431-78c7b33ab776     Repository Update                                                           December 9, 2021 3:54:59 PM CET     Success

odacli describe-component | grep -v ^$
System Version
---------------
19.13.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.13.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.12.0.0.0           19.13.0.0.0
GI                                        19.12.0.0.210720      19.13.0.0.211019
DB                                        19.12.0.0.210720      19.13.0.0.211019
DCSCONTROLLER                             19.13.0.0.0           up-to-date
DCSCLI                                    19.13.0.0.0           up-to-date
DCSAGENT                                  19.13.0.0.0           up-to-date
DCSADMIN                                  19.13.0.0.0           up-to-date
OS                                        7.9                   up-to-date
ILOM                                      5.0.2.24.r141466      up-to-date
BIOS                                      52050300              up-to-date
SHARED CONTROLLER FIRMWARE                VDV1RL04              up-to-date
LOCAL DISK FIRMWARE                       1132                  up-to-date
SHARED DISK FIRMWARE                      1132                  up-to-date
HMP                                       2.4.8.0.600           up-to-date

This update will be limited to Oracle software, so it shouldn’t last hours.

Pre-patching report

Let’s check if patching has the green light:

odacli create-prepatchreport -s -v 19.13.0.0.0

Job details
----------------------------------------------------------------
                     ID:  3e9c9d23-948e-4dbb-8b23-8ab2f6e9fb14
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
                 Status:  Created
                Created:  December 9, 2021 4:26:59 PM CET
                Message:  Use 'odacli describe-prepatchreport -i 3e9c9d23-948e-4dbb-8b23-8ab2f6e9fb14' to check details of results

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------


odacli describe-prepatchreport -i 3e9c9d23-948e-4dbb-8b23-8ab2f6e9fb14

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  3e9c9d23-948e-4dbb-8b23-8ab2f6e9fb14
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
                 Status:  SUCCESS
                Created:  December 9, 2021 4:26:59 PM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
dbi-oda-x8

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.13.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.13.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Patch already applied
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.13.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

Prepatch lasted about 10 minutes on my ODA, and everything seems to be OK.

Patching infrastructure and GI

Let’s start the update-server:

odacli update-server -v 19.13.0.0.0
odacli describe-job -i f0915a98-5acf-4697-94e2-8c0192bcfce2

Job details
----------------------------------------------------------------
                     ID:  f0915a98-5acf-4697-94e2-8c0192bcfce2
            Description:  Server Patching
                 Status:  Success
                Created:  December 10, 2021 2:07:20 PM CET
                Message:  Successfully patched GI with RHP

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validating GI user metadata              December 10, 2021 2:07:33 PM CET    December 10, 2021 2:07:33 PM CET    Success
Creating repositories using yum          December 10, 2021 2:07:34 PM CET    December 10, 2021 2:07:38 PM CET    Success
Updating YumPluginVersionLock rpm        December 10, 2021 2:07:38 PM CET    December 10, 2021 2:07:38 PM CET    Success
Applying OS Patches                      December 10, 2021 2:07:38 PM CET    December 10, 2021 2:16:03 PM CET    Success
Creating repositories using yum          December 10, 2021 2:16:03 PM CET    December 10, 2021 2:16:04 PM CET    Success
Applying HMP Patches                     December 10, 2021 2:16:04 PM CET    December 10, 2021 2:16:22 PM CET    Success
Patch location validation                December 10, 2021 2:16:22 PM CET    December 10, 2021 2:16:22 PM CET    Success
oda-hw-mgmt upgrade                      December 10, 2021 2:16:22 PM CET    December 10, 2021 2:16:55 PM CET    Success
OSS Patching                             December 10, 2021 2:16:55 PM CET    December 10, 2021 2:16:55 PM CET    Success
Checking Ilom patch Version              December 10, 2021 2:16:55 PM CET    December 10, 2021 2:16:56 PM CET    Success
Patch location validation                December 10, 2021 2:16:56 PM CET    December 10, 2021 2:16:56 PM CET    Success
Save password in Wallet                  December 10, 2021 2:16:56 PM CET    December 10, 2021 2:16:56 PM CET    Success
Disabling IPMI v2                        December 10, 2021 2:16:56 PM CET    December 10, 2021 2:16:57 PM CET    Success
Apply Ilom patch                         December 10, 2021 2:16:57 PM CET    December 10, 2021 2:16:57 PM CET    Success
Copying Flash Bios to Temp location      December 10, 2021 2:16:57 PM CET    December 10, 2021 2:16:57 PM CET    Success
Starting the clusterware                 December 10, 2021 2:16:57 PM CET    December 10, 2021 2:20:08 PM CET    Success
registering image                        December 10, 2021 2:20:08 PM CET    December 10, 2021 2:20:08 PM CET    Success
registering working copy                 December 10, 2021 2:20:08 PM CET    December 10, 2021 2:20:08 PM CET    Success
registering image                        December 10, 2021 2:20:08 PM CET    December 10, 2021 2:20:08 PM CET    Success
Creating GI home directories             December 10, 2021 2:20:08 PM CET    December 10, 2021 2:20:08 PM CET    Success
Extract GI clone                         December 10, 2021 2:20:08 PM CET    December 10, 2021 2:20:08 PM CET    Success
Provisioning Software Only GI with RHP   December 10, 2021 2:20:08 PM CET    December 10, 2021 2:20:08 PM CET    Success
Patch GI with RHP                        December 10, 2021 2:20:08 PM CET    December 10, 2021 2:29:39 PM CET    Success
Updating GIHome version                  December 10, 2021 2:29:39 PM CET    December 10, 2021 2:29:42 PM CET    Success
Validate GI availability                 December 10, 2021 2:29:49 PM CET    December 10, 2021 2:29:49 PM CET    Success
Patch KVM CRS type                       December 10, 2021 2:29:49 PM CET    December 10, 2021 2:31:53 PM CET    Success
Update System version                    December 10, 2021 2:31:53 PM CET    December 10, 2021 2:31:53 PM CET    Success
Cleanup JRE Home                         December 10, 2021 2:31:53 PM CET    December 10, 2021 2:31:53 PM CET    Success
Add SYSNAME in Env                       December 10, 2021 2:31:53 PM CET    December 10, 2021 2:31:53 PM CET    Success
Setting ACL for disk groups              December 10, 2021 2:31:53 PM CET    December 10, 2021 2:31:57 PM CET    Success
Update lvm.conf file                     December 10, 2021 2:33:30 PM CET    December 10, 2021 2:33:30 PM CET    Success
Update previous workarounds              December 10, 2021 2:33:30 PM CET    December 10, 2021 2:33:30 PM CET    Success
preRebootNode Actions                    December 10, 2021 2:33:30 PM CET    December 10, 2021 2:36:52 PM CET    Success
Reboot Ilom                              December 10, 2021 2:36:52 PM CET    December 10, 2021 2:36:52 PM CET    Success

Server reboots 5 minutes after the patch ends. On my X8-2M this server patching lasted 30 minutes.

Let’s check the components’ versions now:

odacli describe-component | grep -v ^$ 
System Version
---------------
19.13.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.13.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.13.0.0.0           up-to-date
GI                                        19.13.0.0.211019      up-to-date
DB                                        19.12.0.0.210720      19.13.0.0.211019
DCSCONTROLLER                             19.13.0.0.0           up-to-date
DCSCLI                                    19.13.0.0.0           up-to-date
DCSAGENT                                  19.13.0.0.0           up-to-date
DCSADMIN                                  19.13.0.0.0           up-to-date
OS                                        7.9                   up-to-date
ILOM                                      5.0.2.24.r141466      up-to-date
BIOS                                      52050300              up-to-date
SHARED CONTROLLER FIRMWARE                VDV1RL04              up-to-date
LOCAL DISK FIRMWARE                       1132                  up-to-date
SHARED DISK FIRMWARE                      1132                  up-to-date
HMP                                       2.4.8.0.600           up-to-date

This looks fine, but please check acfs volumes after reboot!!!

In my particular case, I had problem with acfs volumes not mounting after reboot, even if I tried manually:

mount.acfs -o all
mount.acfs: CLSU-00107: operating system function: open64; failed with error data: 1; at location: OOF_1
mount.acfs: CLSU-00101: operating system error message: Operation not permitted
mount.acfs: CLSU-00104: additional error information: open64 (/dev/asm/commonstore-175)
mount.acfs: ACFS-02017: Failed to open volume /dev/asm/commonstore-175. Verify the volume exists.
mount.acfs: ACFS-02014: Mount of /opt/oracle/dcs/commonstore failed.  Error -1 was returned.
mount.acfs: CLSU-00107: operating system function: open64; failed with error data: 1; at location: OOF_1
mount.acfs: CLSU-00101: operating system error message: Operation not permitted
mount.acfs: CLSU-00104: additional error information: open64 (/dev/asm/acfsclone-175)
mount.acfs: ACFS-02017: Failed to open volume /dev/asm/acfsclone-175. Verify the volume exists.
mount.acfs: ACFS-02014: Mount of /opt/oracle/oak/pkgrepos/orapkgs/clones failed.  Error -1 was returned.
...

I had to reinstall the drivers from the new GI home, and reboot again:

/u01/app/19.13.0.0/grid/bin/crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dbi-oda-x8'
CRS-2673: Attempting to stop 'ora.crsd' on 'dbi-oda-x8'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'dbi-oda-x8'
CRS-2673: Attempting to stop 'ora.cvu' on 'dbi-oda-x8'
CRS-2673: Attempting to stop 'ora.DATA.ACFSCLONE.advm' on 'dbi-oda-x8'
CRS-2673: Attempting to stop 'ora.DATA.CESAREVMS.advm' on 'dbi-oda-x8'
...
CRS-4133: Oracle High Availability Services has been stopped.

/u01/app/19.13.0.0/grid/bin/acfsroot install
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9294: updating file /etc/sysconfig/oracledrivers.conf
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9294: updating file /etc/sysconfig/oracledrivers.conf
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9154: Loading 'oracleoks.ko' driver.
ACFS-9154: Loading 'oracleadvm.ko' driver.
ACFS-9154: Loading 'oracleacfs.ko' driver.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9309: ADVM/ACFS installation correctness verified.


/u01/app/19.13.0.0/grid/bin/crsctl start crs

/u01/app/19.13.0.0/grid/bin/acfsroot enable

reboot

Once done, everything was back to normal.

Patching the storage

Patching the storage is only needed for older ODAs/patch levels. In case you need to apply a patch on the storage it’s easy: there is a pre-patch, and then the patch:

odacli create-prepatchreport -st -v 19.13.0.0.0
odacli update-storage -v 19.13.0.0.0

For HA ODAs using RAC, patching can be done in a rolling fashion:

odacli update-storage -v 19.13.0.0.0 --rolling

I never encountered troubles during storage patching, so it should be fine.

Patching the DB homes

Since 19.11, DB homes are created on an acfs filesystem. If you come from 19.9 or 19.10, you will need to configure this filesystem:

odacli configure-dbhome-storage -dg DATA

Time for patching the DB homes depends on the number of DB homes and number of databases. In this example, 2 DB homes are deployed:

odacli list-dbhomes

ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
721c1fc8-b394-4682-b230-21a197e997b3     OraDB19000_home10    19.12.0.0.210720                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_10 CONFIGURED
2e702142-048e-42a2-a570-82318458f72d     OraDB19000_home11    19.12.0.0.210720                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_11 CONFIGURED

A prepatching is also needed here, for example if I want to patch the second home:

odacli create-prepatchreport -d -i 2e702142-048e-42a2-a570-82318458f72d -v 19.13.0.0.0
odacli describe-prepatchreport -i 2f5547b1-e95e-489f-bd68-aba0c4260765

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  2f5547b1-e95e-489f-bd68-aba0c4260765
            Description:  Patch pre-checks for [DB, ORACHKDB]: DbHome is OraDB19000_home11
                 Status:  SUCCESS
                Created:  December 10, 2021 6:12:44 PM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
dbi-oda-x8

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__DB__
Validate DB Home ID             Success   Validated DB Home ID:
                                          2e702142-048e-42a2-a570-82318458f72d
Validate patching tag           Success   Validated patching tag: 19.13.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate minimum agent version  Success   Validated minimum agent version
Is GI upgraded                  Success   Validated GI is upgraded
Validate available space for    Success   Validated free space required under
db                                        /u01/app/odaorahome
Validate dbHomesOnACFS          Success   User has configured diskgroup for
configured                                Database homes on ACFS
Validate Oracle base            Success   Successfully validated Oracle Base
Is DB clone available           Success   Successfully validated clone file
                                          exists
Evaluate DBHome patching with   Success   Successfully validated updating
RHP                                       dbhome with RHP
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

Now I can apply the patch on this DB home:

odacli update-dbhome -i 2e702142-048e-42a2-a570-82318458f72d -v 19.13.0.0.0 
odacli describe-job -i ca7673d2-3dd7-45b8-b3e4-9e818b20a4cb

Job details
----------------------------------------------------------------
                     ID:  ca7673d2-3dd7-45b8-b3e4-9e818b20a4cb
            Description:  DB Home Patching: Home Id is 2e702142-048e-42a2-a570-82318458f72d
                 Status:  Success
                Created:  December 10, 2021 6:27:49 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Adding USER SSH_EQUIVALENCE              December 10, 2021 6:27:54 PM CET    December 10, 2021 6:27:54 PM CET    Success
Adding USER SSH_EQUIVALENCE              December 10, 2021 6:27:54 PM CET    December 10, 2021 6:27:54 PM CET    Success
Adding USER SSH_EQUIVALENCE              December 10, 2021 6:27:54 PM CET    December 10, 2021 6:27:55 PM CET    Success
Creating wallet for DB Client            December 10, 2021 6:28:32 PM CET    December 10, 2021 6:28:32 PM CET    Success
Patch databases by RHP                   December 10, 2021 6:28:32 PM CET    December 10, 2021 6:33:15 PM CET    Success
updating database metadata               December 10, 2021 6:33:15 PM CET    December 10, 2021 6:33:15 PM CET    Success
Set log_archive_dest for Database        December 10, 2021 6:33:15 PM CET    December 10, 2021 6:33:18 PM CET    Success
Update System version                    December 10, 2021 6:33:18 PM CET    December 10, 2021 6:33:18 PM CET    Success
TDE parameter update                     December 10, 2021 6:33:45 PM CET    December 10, 2021 6:33:45 PM CET    Success

A new DB home has been created and my database is linked to this new one:

odacli list-dbhomes

ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
721c1fc8-b394-4682-b230-21a197e997b3     OraDB19000_home10    19.12.0.0.210720                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_10 CONFIGURED
2e702142-048e-42a2-a570-82318458f72d     OraDB19000_home11    19.12.0.0.210720                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_11 CONFIGURED
abe18169-18e3-4a3f-a7c8-8c1443863910     OraDB19000_home15    19.13.0.0.211019                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_15 CONFIGURED

odacli list-databases | grep abe18169

617de33b-d9b4-4142-89a5-6bce21e8c00b     DBITSTEE   SI       19.13.0.0.211019     false      OLTP     odb2     ASM        CONFIGURED   abe18169-18e3-4a3f-a7c8-8c1443863910

The old DB home can be removed safely:

odacli delete-dbhome -i 2e702142-048e-42a2-a570-82318458f72d

For each database running in that DB home, a parameter needs to be changed:

su - oracle
. oraenv <<< DBITSTEE
sqlplus / as sysdba
alter system set "_enable_numa_support"=true scope=spfile sid='*';
exit
srvctl stop database -d DBITSTEE_SITE1
srvctl start database -d DBITSTEE_SITE1

This only concerns multi-processor ODAs (not S ones) and it will force an instance to use local memory modules, those associated to the processor where the instance is running. This should improve overall performance.

Patching the other DB homes is done the same way.

Remember that patching the standby databases will raise an error, as datapatch cannot be applied on a mounted or read only database. Patch should be applied on primary and it will then be applied automatically on standby as it’s related to database objects.

Please check my previous blog post about patching ODAs in Data Guard environment.

I would recommand to check on each primary the patch level after patching each DB home:

su – oracle
. oraenv <<< DBITSTEE
sqlplus / as sysdba
set serverout on
exec dbms_qopatch.get_sqlpatch_status;
Patch Id : 32876380
	Action : APPLY
	Action Time : 03-AUG-2021 16:15:57
	Description : OJVM RELEASE UPDATE: 19.12.0.0.210720 (32876380)
	Logfile :
/u01/app/oracle/cfgtoollogs/sqlpatch/32876380/24269510/32876380_apply_G17741_CDB
ROOT_2021Aug03_15_59_00.log
	Status : SUCCESS

Patch Id : 32904851
	Action : APPLY
	Action Time : 03-AUG-2021 16:15:57
	Description : Database Release Update : 19.12.0.0.210720 (32904851)
	Logfile :
/u01/app/oracle/cfgtoollogs/sqlpatch/32904851/24343243/32904851_apply_G17741_CDB
ROOT_2021Aug03_15_59_01.log
	Status : SUCCESS

Patch Id : 32876380
	Action : ROLLBACK
	Action Time : 10-DEC-2021 18:33:12
	Description : OJVM RELEASE UPDATE: 19.12.0.0.210720 (32876380)
	Logfile :
/u01/app/odaorabase/oracle/cfgtoollogs/sqlpatch/32876380/24269510/32876380_rollb
ack_DBITSTEE_2021Dec10_18_31_06.log
	Status : SUCCESS

Patch Id : 33192694
	Action : APPLY
	Action Time : 10-DEC-2021 18:33:15
	Description : OJVM RELEASE UPDATE: 19.13.0.0.211019 (33192694)
	Logfile :
/u01/app/odaorabase/oracle/cfgtoollogs/sqlpatch/33192694/24421575/33192694_apply
_DBITSTEE_2021Dec10_18_31_33.log
	Status : SUCCESS

Patch Id : 33192793
	Action : APPLY
	Action Time : 10-DEC-2021 18:33:15
	Description : Database Release Update : 19.13.0.0.211019 (33192793)
	Logfile :
/u01/app/odaorabase/oracle/cfgtoollogs/sqlpatch/33192793/24462514/33192793_apply
_DBITSTEE_2021Dec10_18_31_33.log
	Status : SUCCESS

PL/SQL procedure successfully completed.
exit

Final checks

Let’s get the final versions:

odacli describe-component | grep -v ^$
System Version
---------------
19.13.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.13.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.13.0.0.0           up-to-date
GI                                        19.13.0.0.211019      up-to-date
DB {
[ OraDB19000_home10 ]                     19.12.0.0.210720      19.13.0.0.211019
[ OraDB19000_home15 ]                     19.13.0.0.211019      up-to-date
}
DCSCONTROLLER                             19.13.0.0.0           up-to-date
DCSCLI                                    19.13.0.0.0           up-to-date
DCSAGENT                                  19.13.0.0.0           up-to-date
DCSADMIN                                  19.13.0.0.0           up-to-date
OS                                        7.9                   up-to-date
ILOM                                      5.0.2.24.r141466      up-to-date
BIOS                                      52050300              up-to-date
SHARED CONTROLLER FIRMWARE                VDV1RL04              up-to-date
LOCAL DISK FIRMWARE                       1132                  up-to-date
SHARED DISK FIRMWARE                      1132                  up-to-date
HMP                                       2.4.8.0.600           up-to-date

One of my DB home is still running 19.12 because I didn’t patch it.

Cleanse the old patches

The old patches will never be used again. For this ODA, history was:

Deploy = 19.10.0.0.0 => Patch 19.12.0.0.0 => Patch 19.13.0.0.0

If you don’t remember the history of your ODA, have a look in this folder:

ls -lrt /opt/oracle/oak/pkgrepos/System/
total 44
-rwxr-xr-x. 1 root root 26255 Mar 18  2021 system_repos_metadata.xml
drwxr-xr-x  2 root root  4096 Aug 27 15:18 19.12.0.0.0
drwxr-xr-x. 5 root root  4096 Nov  3 09:56 19.10.0.0.0
drwxrwxr-x  2 root root  4096 Nov 27 17:16 19.13.0.0.0
drwxr-xr-x. 2 root root  4096 Dec  9 15:54 latest

You can presume this ODA was deployed with 19.10 and first patched with 19.12.

Let’s remove the previous patch:

odacli cleanup-patchrepo -cl -comp db,gi -v 19.12.0.0.0

Put back your own settings

Once everything is OK, don’t forget to put back your settings:

  • add your additional rpms manually if needed
  • put back your profile scripts for grid and oracle users
  • …

And what about DB Systems update?

If you use DB Systems on your ODA, meaning that some of your databases are running in dedicated VMs, the patch is not applied inside the DB System.

Let’s do a describe component in the VM itself:

odacli describe-component | grep -v ^$
System Version
---------------
19.12.0.0.0
System node Name
---------------
srvdb02
Local System Version
---------------
19.12.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.12.0.0.0           19.13.0.0.0
GI                                        19.12.0.0.210720      19.13.0.0.211019
DB                                        19.12.0.0.210720      19.13.0.0.211019
DCSCONTROLLER                             19.12.0.0.0           19.13.0.0.0
DCSCLI                                    19.12.0.0.0           19.13.0.0.0
DCSAGENT                                  19.12.0.0.0           19.13.0.0.0
DCSADMIN                                  19.12.0.0.0           19.13.0.0.0
OS                                        7.9                   up-to-date

Patching a DB System is done being connected to it, and commands are similar to what you’ve done on bare metal.

odacli update-dcsadmin -v 19.13.0.0.0
odacli update-dcscomponents -v 19.13.0.0.0
odacli update-dcsagent -v 19.13.0.0.0
odacli create-prepatchreport -s -v 19.13.0.0.0
odacli describe-prepatchreport -i 584efe26-ff23-4024-a1e9-8182b2b38a89
odacli update-server -v 19.13.0.0.0
odacli create-prepatchreport -d -i b1a0bfd9-5db5-4bce-8e4d-49f07480cc4e -v 19.13.0.0.0
odacli describe-prepatchreport -i 03b90bd8-a0b5-4998-83ba-c3943917d951
odacli update-dbhome -i b1a0bfd9-5db5-4bce-8e4d-49f07480cc4e -v 19.13.0.0.0
odacli delete-dbhome -i b1a0bfd9-5db5-4bce-8e4d-49f07480cc4e
odacli describe-component | grep -v ^$
System Version
---------------
19.13.0.0.0
System node Name
---------------
srvdb02
Local System Version
---------------
19.13.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.13.0.0.0           up-to-date
GI                                        19.13.0.0.211019      up-to-date
DB                                        19.13.0.0.211019      up-to-date
DCSCONTROLLER                             19.13.0.0.0           up-to-date
DCSCLI                                    19.13.0.0.0           up-to-date
DCSAGENT                                  19.13.0.0.0           up-to-date
DCSADMIN                                  19.13.0.0.0           up-to-date
OS                                        7.9                   up-to-date

This seems obvious but you cannot apply a patch on a DB System if the host is not up to date: 

Validate BM versions     Failed    Operation: Update Server failed, node dbi-oda-x8 is not running with the supported version for OAK,GI.

Using DB Systems is a nice option if you want to keep things clean and isolated. But it’s much more work when it comes to patching, as it’s not much faster to patch a DB System compared to bare metal.

Conclusion

My ODA is now in the latest 19c version. This was not too difficult coming from a recent version, it could be more challenging if you come from an older one.

L’article Oracle Database Appliance 19.13: what’s new and how to patch? est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/oracle-database-appliance-19-13-whats-new-and-how-to-patch/feed/ 0 ODA 19.12: How can you manage DB Systems with odacli? https://www.dbi-services.com/blog/oda-19-12-how-can-you-manage-db-systems-with-odacli/ https://www.dbi-services.com/blog/oda-19-12-how-can-you-manage-db-systems-with-odacli/#comments Fri, 26 Nov 2021 15:55:05 +0000 https://www.dbi-services.com/blog/oda-19-12-how-can-you-manage-db-systems-with-odacli/ Introduction As virtualization is available for all ODAs now, you could ask yourself if it’s a good idea to use this virtualization or not. Databases can be created as bare metal, but also within a DB System for better isolation. And both can be used on the same ODA. If you’re looking for how to […]

L’article ODA 19.12: How can you manage DB Systems with odacli? est apparu en premier sur dbi Blog.

]]> Introduction

As virtualization is available for all ODAs now, you could ask yourself if it’s a good idea to use this virtualization or not. Databases can be created as bare metal, but also within a DB System for better isolation. And both can be used on the same ODA.

If you’re looking for how to deploy an ODA with KVM virtualization, please have a look at my previous blogpost here.

What is a DB System?

A DB System is an OCI (the public Cloud from Oracle) feature brought to the ODA. This is a single component that includes a VM with a Linux OS, a DB home and a database. On ODA, you don’t really need that because databases are most of the time configured as bare metal ones (running on the server itself), but it’s now possible to configure DB System as well. Appart for better isolation (for example, separating PROD from DEV databases on the same appliance), this is also needed for network segregation: it’s possible to associate different DB Systems to different VLANs. And if you need to tweak a database, it does not impact the other DB Systems.

The drawbacks I could see are for sure more complexity and more systems to manage. Use bare metal only if you want to keep it simple.

Creating a DB System

For creating a DB System, you will need a template (provided aside the 19.12 patch), and corresponding GI and DB clones:

cd /opt/dbi
unzip p33152237_1912000_Linux-x86-64.zip
odacli update-repository -f /opt/dbi/odacli-dcs-19.12.0.0.0-ODAVM.zip
odacli describe-dbsystem-image
DB System Image details
--------------------------------------------------------------------------------
Component Name        Supported Versions    Available Versions
--------------------  --------------------  --------------------

DBVM                  19.12.0.0.0           19.12.0.0.0

GI                    19.12.0.0.210720      19.12.0.0.210720
                      21.3.0.0.210720       not-available

DB                    19.12.0.0.210720      19.12.0.0.210720
                      21.3.0.0.210720       not-available

Let’s use 19c version in this example, as 21c is an innovation release.

A DB System will need a CPU pool (it can be shared across multiple DB Systems) and a json file with various parameters (mine is provided at the end):

odacli create-cpupool -n cpupool4dbsystems -c 4 -dbs
odacli create-dbsystem -p /opt/dbi/create_dbsystem_srvdb01.json

DB System creation lasts 30 minutes, and this one has an odb2 shape.

Note these 2 points:
– The shape for the VM and for the database need to be the same, or you will get an error

DCS-10045:Validation error encountered: The DB System shape and Database shape should be the same.

– Multiple DB Systems cannot have the same database name. It would normally be possible, but not on ODA.

DCS-10044:Object dbName already exists with same value VIRTDB.

odacli to manage DB Systems

You can list the DB systems configured on your ODA:

odacli list-dbsystems

Name                  Shape       Cores  Memory      GI version          DB version          Status           Created                  Updated
--------------------  ----------  -----  ----------  ------------------  ------------------  ---------------  -----------------------  -----------------------
srvdb01               odb2        4      16.00 GB    19.12.0.0.210720    19.12.0.0.210720    FAILED           2021-11-25 16:04:05 CET  2021-11-25 16:22:15 CET

And as DB Systems are VMs, stop and start commands are available :

odacli stop-dbsystem -n srvdb01
odacli start-dbsystem -n srvdb01

You can also have extended information about a DB System, for example its current status:

odacli describe-dbsystem -n srvdb01 | grep State
             Target State:  ONLINE
            Current State:  ONLINE

Another example, the shape of the VM and database:

odacli describe-dbsystem -n srvdb01 | grep Shape
                    Shape:  odb2
                    Shape:  odb2

You can change the shape of a DB System (but only going up is allowed):

odacli modify-dbsystem -n srvdb01 -s odb4

odacli list-dbsystems
Name                  Shape       Cores  Memory      GI version          DB version          Status           Created                  Updated
--------------------  ----------  -----  ----------  ------------------  ------------------  ---------------  -----------------------  -----------------------
srvdb01               odb4        4      32.00 GB    19.12.0.0.210720    19.12.0.0.210720    CONFIGURED       2021-11-24 19:02:04 CET  2021-11-25 14:58:49 CET

Your DB System will need a restart to correctly apply the shape to the database.

Basically, you can also change network settings and cpu pool association.

For sure you can also delete a db system (after stopping it):

odacli delete-dbsystem -n srvdb01

odacli inside the DB System

The purpose of a DB System is to host a single database. odacli is then limited inside a DB System.

So what can we do? Displaying information about database and dbhome is OK:

odacli list-databases

ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
37b7becf-7697-4ea8-9dd7-d91cfb36444a     VIRTDB     SI       19.12.0.0.210720     false      OLTP     odb4     ASM        CONFIGURED   7916c735-1503-4fb4-b414-3dcdb4c68517

odacli list-dbhomes

ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
6dabc902-cc0e-4fa6-82a7-9fa1f6361599     OraDB19000_home1     19.12.0.0.210720                         /u01/app/oracle/product/19.0.0.0/dbhome_1     CONFIGURED

Creating a new dhbome is also OK, but not creating a database:

odacli create-dbhome -v 19.12.0.0.210720
odacli list-dbhomes

ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
6dabc902-cc0e-4fa6-82a7-9fa1f6361599     OraDB19000_home1     19.12.0.0.210720                         /u01/app/oracle/product/19.0.0.0/dbhome_1     CONFIGURED
b1a0bfd9-5db5-4bce-8e4d-49f07480cc4e     OraDB19000_home2     19.12.0.0.210720                         /u01/app/oracle/product/19.0.0.0/dbhome_2     CONFIGURED


odacli create-database -u VIRTDB2 -dh 'b1a0bfd9-5db5-4bce-8e4d-49f07480cc4e' -n VIRTDB2 -s odb1 -r asm
DCS-10001:Internal error encountered: Available Memory is less than SGA Size { Available : 796mb and SGA Size : 7782mb }.

There is not enough memory for another instance, let’s decrease the shape of the previous database and retry:

odacli modify-database -i 37b7becf-7697-4ea8-9dd7-d91cfb36444a -s odb1

odacli list-databases

ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
37b7becf-7697-4ea8-9dd7-d91cfb36444a     VIRTDB     SI       19.12.0.0.210720     false      OLTP     odb1     ASM        CONFIGURED   6dabc902-cc0e-4fa6-82a7-9fa1f6361599

Let’s retry:

odacli create-database -u VIRTDB2 -dh '3a6e44cc-6f22-4696-b84f-55affdc5e09b' -n VIRTDB2 -s odb1 -r asm
odacli list-databases

ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
37b7becf-7697-4ea8-9dd7-d91cfb36444a     VIRTDB     SI       19.12.0.0.210720     false      OLTP     odb1     ASM        CONFIGURED   6dabc902-cc0e-4fa6-82a7-9fa1f6361599
3270f020-afd6-4658-a029-754b48832d3b     VIRTDB2    SI       19.0.0.0             false      OLTP     odb1     ASM        FAILED       b1a0bfd9-5db5-4bce-8e4d-49f07480cc4e

Actually, it does not work because of an internal error:

odacli describe-job -i ecdda655-cbac-4782-aa88-f6a6b5eb5960 | grep Message
                Message:  DCS-10001:Internal error encountered: Failed to create the database VIRTDB2.

Let’s remove the first database and create another one, it may work:

odacli delete-database -i 37b7becf-7697-4ea8-9dd7-d91cfb36444a
odacli delete-database -i 3270f020-afd6-4658-a029-754b48832d3b
odacli create-database -u VIRTDB2 -dh '404a58c6-913e-4c6d-bdb1-f0f99b684fe2' -n VIRTDB2 -s odb1 -r asm
odacli list-databases

ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
6f2ec94b-7a79-4ea4-9d0a-d6fc613ddcfa     VIRTDB2    SI       19.12.0.0.210720     false      OLTP     odb1     ASM        CONFIGURED   b1a0bfd9-5db5-4bce-8e4d-49f07480cc4e

Yes it works. But the shape looks like odb4, not odb1:

[root@srvdb01 ~]# su - oracle
Last login: Wed Nov 24 18:44:18 CET 2021
[oracle@srvdb01 ~]$ . oraenv <<< VIRTDB2
ORACLE_SID = [oracle] ? The Oracle base has been set to /u01/app/oracle
[oracle@srvdb01 ~]$ sqlplus -s / as sysdba
sho parameter sga_

NAME				     TYPE	 VALUE
------------------------------------ ----------- ------------------------------
sga_max_size			     big integer 7792M
sga_min_size			     big integer 0
sga_target			     big integer 7792M
unified_audit_sga_queue_size	     integer	 1048576

Managing databases on a DB System is not very clean. You can do things, but I would not recommend.

Features linked to the database created during provisioning are OK, for example applying a backup configuration for your database:

odacli create-backupconfig -n nfsBackupConfig -w 7 -d NFS -c /backup/srvdb01

odacli modify-database -in VIRTDB2 -bin nfsBackupConfig

odacli list-schedules | grep -e VIRTDB2 -e ID -e ---
ID                                       Name                      Description                                        CronExpression                 Disabled
---------------------------------------- ------------------------- -------------------------------------------------- ------------------------------ --------
86211dc2-af0c-4e21-a8b6-58df85d0fd1f     database_backup_6f2ec94b-7a79-4ea4-9d0a-d6fc613ddcfa backup database : VIRTDB2                          0 24 5 1/1 * ? *               false
e49d5605-d1f2-47ea-9d6e-f266fb5d724b     archive_log_backup_6f2ec94b-7a79-4ea4-9d0a-d6fc613ddcfa backup archive logs : VIRTDB2                      0 0/30 * ? * * *               false

Backups are now scheduled for my database on this DB System, with a 7-day retention on a nfs share.

Conclusion

DB Systems should be seen as single objects and managed at the DB System level mainly. If you need to change something locally on the database it’s still possible but odacli may not help.

Demo json file

create_dbsystem_srvdb01.json

{
    "system": {
        "name": "srvdb01",
        "shape": "odb2",
        "systemPassword": "**********",
        "timeZone": "Europe/Zurich",
        "diskGroup": "DATA",
        "cpuPoolName": "cpupool4dbsystems",
        "enableRoleSeparation": true,
        "customRoleSeparation": {
            "groups": [
                {
                    "name": "oinstall",
                    "id": 1001,
                    "role": "oinstall"
                },
                {
                    "name": "dbaoper",
                    "id": 1002,
                    "role": "dbaoper"
                },
                {
                    "name": "dba",
                    "id": 1003,
                    "role": "dba"
                },
                {
                    "name": "asmadmin",
                    "id": 1004,
                    "role": "asmadmin"
                },
                {
                    "name": "asmoper",
                    "id": 1005,
                    "role": "asmoper"
                },
                {
                    "name": "asmdba",
                    "id": 1006,
                    "role": "asmdba"
                }
            ],
            "users": [
                {
                    "name": "grid",
                    "id": 1000,
                    "role": "gridUser"
                },
                {
                    "name": "oracle",
                    "id": 1001,
                    "role": "oracleUser"
                }
            ]
        }
    },
    "database": {
        "name": "VIRTDB",
        "uniqueName": "VIRTDB",
        "domainName": "dbi-lab.ch",
        "adminPassword": "********",
        "version": "19.12.0.0.210720",
        "edition": "SE",
        "type": "SI",
        "dbClass": "OLTP",
        "shape": "odb2",
        "role": "PRIMARY",
        "targetNodeNumber": null,
        "enableDbConsole": false,
        "enableUnifiedAuditing": true,
        "redundancy" : "MIRROR",
        "characterSet": {
            "characterSet": "AL32UTF8",
            "nlsCharacterset": "AL16UTF16",
            "dbTerritory": "AMERICA",
            "dbLanguage": "ENGLISH"
        },
        "rmanBackupPassword": null,
        "enableTDE": false,
        "isCdb": false
    },
    "network": {
        "domainName": "dbi-lab.ch",
        "ntpServers": ["21.39.35.10"],
        "dnsServers": [
            "8.8.8.8","8.8.4.4"
        ],
        "nodes": [
            {
                "name": "srvdb01",
                "ipAddress": "10.36.10.242",
                "netmask": "255.255.255.0",
                "gateway": "10.36.10.1",
                "number": 0
            }
        ],
  "publicVNetwork": "pubnet"
    },
    "grid": {
        "language": "en"
    }
}

L’article ODA 19.12: How can you manage DB Systems with odacli? est apparu en premier sur dbi Blog.

]]> https://www.dbi-services.com/blog/oda-19-12-how-can-you-manage-db-systems-with-odacli/feed/ 1