I wrote this article to show how you can integrate Legacy OEM with your Cloud environment in this case Oracle ExaCC.To have overall view on your Exadata X9M Cloud Infrastructure, statistics related to CPU/Memory/ASM Storage usage. Your on premise monitoring OEM can read data from Oracle cloud using OCI libraries and present them in combination with your standard DB/VMs target monitoring, this give you powerful tool for overall view on your Enterprise environment.

Since version OEM release update 13.5.0.23 of OMS / Agents. There is few new interesting Dashboards and panels related to OCI Exadata Infrastructure, also RU23 introduces the Oracle Enterprise Manager Cloud Control Exadata Management Pack and Zero Data Loss Recovery Appliance (ZDLRA) Management Pack, some Dashboards, panels view below:

Biggest advantage of this integration is that you will have wider view to your cloud environment, Concept of ExaCC is that you see and have access to VMs and databases, hardware of Exadata Dbservers and Cellservers are normally managed by Oracle.

Adding this OCI integration to OEM give you viasability of physical cell servers and their statistics.

Now lets move on how this can be configured

Prerequisites:

1. OEM OMS/Agents update to minimum 13.5.0.23 (AUG 07/2024), why? since this version there is OCI integration enhancements and new dashboards,management packs for ExaCC. I described this in another post link here: https://www.dbi-services.com/blog/oem-cloud-control-release-update-to-13-5-0-23
2. Get your Exacli user/password to cellservers from your environment , in procedure I will describe how to do it
3. Have OCI client/libraries installed on one of Agents host.
4. Have access to OCI directly or over HTTPS proxy from Agent host.

Step by Step Procedure:

Get Exacli username and password to your ExaCC cellservers(this step is optional but recommend to have more statistics from cell servers.

Login to one of yours Virtual Exadata vm server and check username and password for StorageCells.

username usually have format: cloud_user_<crs_cluster_name>

to get clustername and password ,login as grid user end execute:

[grid@s***n1 ~]$ crsctl get cluster name; 
CRS-6724: Current cluster name is 'zh1234clua1' 

#so in this case username will be cloud_user_zh1234clua1

#now get Exadata initial password
[opc@s***n1 ~]$ sudo su - 
[root@s***n1 ~]# /opt/exacloud/get_cs_data.py 
*** WARNING : deprecated key derivation used. 
Using -iter or -pbkdf2 would be better. 
ExaCli initial password is: 
 **************************** 


#Collect the storage server IP address from 
/etc/oracle/cell/network-config/cellip.ora on the VM Guest. 

#Test Connection to one of cell IP addresses 
#using exacli provide username and password gathered above 
exacli -l cloud_user_zh1234clua1 -c 100.100.0.13 -e list physicaldisk 
Password: **************************** 

#in output you should receive disk list from Storage cell , if not then means that your username or passwords are incorrect.

Save credentials for later.

Create new named credentials in OEM:

Go to Setup > Security > Named Credentials and create new with user and passwords saved above:

Now we have connection between OEM and storage Cellservers.

Create user/group/policy for monitoring and assign proper policy’s in OCI.

Login to OCI and create new user,group and policys

Go to Identity > Domains > Groups and create new (use name which apply to your naming convention

Create New user and attache to Group created earler:

Create API keys:

Copy OCI configuration config file and save ssh key.

Create new policy with 2 Statements (privileges to be able to read your compartment objects) like below:

Go to Identity & Security > Policies

Download and install OCI Client on your server.

In our case, we installing OCI on OMS server and use EM Agent deployed to this machine. To use OCI API to connect and gather data from Oracle Cloud about our infrastructure using HTTPS_PROXY server.

To install OCI please follow official documentation, there are different options of installation and OS versions: https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm#InstallingCLI__oraclelinux8

In our case we installing OCI Cli as oracle user on Oracle Linux 8

First we set http/s proxy :

oracle@agent1:~/ export HTTPS_PROXY=http://proxy.proxy.com:3038 
oracle@agent1:~/ export HTTP_PROXY=http://proxy.proxy.com:3038 

Download newest OCI libraries :

oracle@agent1:~/# mkdir -p /u01/app/oracle/oci/
oracle@agent1:~/# cd  /u01/app/oracle/oci/
oracle@agent1:/u01/app/oracle/oci/# bash -c "$(curl -L https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.sh)" 

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current 
                                 Dload  Upload   Total   Spent    Left  Speed 
100 16926  100 16926    0     0   199k      0 --:--:-- --:--:-- --:--:--  199k 
    ****************************************************************************** 
    You have started the OCI CLI Installer in interactive mode. If you do not wish 
    to run this in interactive mode, please include the --accept-all-defaults option. 
    If you have the script locally and would like to know more about 
    input options for this script, then you can run: 
    ./install.sh -h 
    If you would like to know more about input options for this script, refer to: 
    https://github.com/oracle/oci-cli/blob/master/scripts/install/README.rst 
    ****************************************************************************** 
Downloading Oracle Cloud Infrastructure CLI install script from https://raw.githubusercontent.com/oracle/oci-cli/v3.2.1/scripts/install/install.py to /tmp/oci_cli_install_tmp_VwnM. 
######################################################################################################################################################################### 100.0% 
Running install script. 
python3 /tmp/oci_cli_install_tmp_VwnM 
-- Verifying Python version. 
-- Python version 3.6.8 okay. 
===> In what directory would you like to place the install? (leave blank to use '/home/oracle/lib/oracle-cli'): 
-- Creating directory '/home/oracle/lib/oracle-cli'. 
-- We will install at '/home/oracle/lib/oracle-cli'. 
===> In what directory would you like to place the 'oci' executable? (leave blank to use '/home/oracle/bin'): 
-- Creating directory '/home/oracle/bin'. 
-- The executable will be in '/home/oracle/bin'. 
===> In what directory would you like to place the OCI scripts? (leave blank to use '/home/oracle/bin/oci-cli-scripts'): 
-- Creating directory '/home/oracle/bin/oci-cli-scripts'. 
-- The scripts will be in '/home/oracle/bin/oci-cli-scripts'. 
===> Currently supported optional packages are: ['db (will install cx_Oracle)'] 
What optional CLI packages would you like to be installed (comma separated names; press enter if you don't need any optional packages)?: 
-- The optional packages installed will be ''. 
-- Trying to use python3 venv. 
-- Executing: ['/usr/bin/python3', '-m', 'venv', '/home/oracle/lib/oracle-cli'] 
-- Executing: ['/home/oracle/lib/oracle-cli/bin/pip', 'install', '--upgrade', 'pip'] 
Collecting pip 
  Downloading https://files.pythonhosted.org/packages/a4/6d/6463d49a933f547439d6b5b98b46af8742cc03ae83543e4d7688c2420f8b/pip-21.3.1-py3-none-any.whl (1.7MB) 
    100% |████████████████████████████████| 1.7MB 847kB/s 
Installing collected packages: pip 
  Found existing installation: pip 9.0.3 
    Uninstalling pip-9.0.3: 
      Successfully uninstalled pip-9.0.3 
Successfully installed pip-21.3.1 
You are using pip version 21.3.1, however version 24.2 is available. 
You should consider upgrading via the 'pip install --upgrade pip' command. 
-- Executing: ['/home/oracle/lib/oracle-cli/bin/pip', 'install', '--cache-dir', '/tmp/tmpk9na7c02', 'wheel', '--upgrade'] 
Collecting wheel 
  Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB) 
Installing collected packages: wheel 
Successfully installed wheel-0.37.1 
-- Executing: ['/home/oracle/lib/oracle-cli/bin/pip', 'install', '--cache-dir', '/tmp/tmpk9na7c02', 'oci_cli', '--upgrade'] 
Collecting oci_cli 
  Downloading oci_cli-3.48.0-py3-none-any.whl (50.7 MB) 
     |████████████████████████████████| 50.7 MB 2.6 MB/s 
Collecting jmespath==0.10.0 
  Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 
Collecting prompt-toolkit==3.0.29 
  Downloading prompt_toolkit-3.0.29-py3-none-any.whl (381 kB) 
     |████████████████████████████████| 381 kB 51.8 MB/s 
Collecting arrow>=1.0.0 
  Donloading arrow-1.2.3-py3-none-any.whl (66 kB) 
     |████████████████████████████████| 66 kB 14.0 MB/s 
Collecting PyYAML<=6.0.1,>=5.4 
  Downloading PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (677 kB) 
     |████████████████████████████████| 677 kB 47.5 MB/s 
Collecting oci==2.134.0 
  Downloading oci-2.134.0-py3-none-any.whl (27.9 MB) 
     |████████████████████████████████| 27.9 MB 96.5 MB/s 
Collecting terminaltables==3.1.10 
  Downloading terminaltables-3.1.10-py2.py3-none-any.whl (15 kB) 
Collecting certifi 
  Downloading certifi-2024.8.30-py3-none-any.whl (167 kB) 
     |████████████████████████████████| 167 kB 106.3 MB/s 
Collecting cryptography<43.0.0,>=3.2.1 
  Downloading cryptography-40.0.2-cp36-abi3-manylinux_2_28_x86_64.whl (3.7 MB) 
     |████████████████████████████████| 3.7 MB 25.3 MB/s 
Collecting six>=1.15.0 
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 
Collecting pyOpenSSL<25.0.0,>=17.5.0 
  Downloading pyOpenSSL-23.2.0-py3-none-any.whl (59 kB) 
     |████████████████████████████████| 59 kB 18.3 MB/s 
Collecting python-dateutil<3.0.0,>=2.5.3 
  Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) 
     |████████████████████████████████| 229 kB 61.2 MB/s 
Collecting pytz>=2016.10 
  Downloading pytz-2024.2-py2.py3-none-any.whl (508 kB) 
     |████████████████████████████████| 508 kB 99.2 MB/s 
Collecting click==8.0.4 
  Downloading click-8.0.4-py3-none-any.whl (97 kB) 
     |████████████████████████████████| 97 kB 19.2 MB/s 
Collecting importlib-metadata 
  Downloading importlib_metadata-4.8.3-py3-none-any.whl (17 kB) 
Collecting circuitbreaker<2.0.0,>=1.3.1 
  Downloading circuitbreaker-1.4.0.tar.gz (9.7 kB) 
  Preparing metadata (setup.py) ... done 
Collecting wcwidth 
  Downloading wcwidth-0.2.13-py2.py3-none-any.whl (34 kB) 
Collecting typing-extensions 
  Downloading typing_extensions-4.1.1-py3-none-any.whl (26 kB) 
Collecting cffi>=1.12 
  Downloading cffi-1.15.1-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (402 kB) 
     |████████████████████████████████| 402 kB 30.7 MB/s 
Collecting pycparser 
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB) 
     |████████████████████████████████| 118 kB 89.7 MB/s 
Collecting zipp>=0.5 
  Downloading zipp-3.6.0-py3-none-any.whl (5.3 kB) 
Building wheels for collected packages: circuitbreaker 
  Building wheel for circuitbreaker (setup.py) ... done 
  Created wheel for circuitbreaker: filename=circuitbreaker-1.4.0-py3-none-any.whl size=7506 sha256=cd24462c8ab8bb71fba04fa996852c0cff6c93749f3a2c74591f3a1cbfa756ec 
  Stored in directory: /tmp/tmpk9na7c02/wheels/46/17/98/db2eb826e4a98da672cffe66ec16838182cde0cf19ad2c0c70 
Successfully built circuitbreaker 
Installing collected packages: pycparser, cffi, zipp, typing-extensions, six, cryptography, wcwidth, pytz, python-dateutil, pyOpenSSL, importlib-metadata, circuitbreaker, certifi, terminaltables, PyYAML, prompt-toolkit, oci, jmespath, click, arrow, oci-cli 
Successfully installed PyYAML-6.0.1 arrow-1.2.3 certifi-2024.8.30 cffi-1.15.1 circuitbreaker-1.4.0 click-8.0.4 cryptography-40.0.2 importlib-metadata-4.8.3 jmespath-0.10.0 oci-2.134.0 oci-cli-3.48.0 prompt-toolkit-3.0.29 pyOpenSSL-23.2.0 pycparser-2.21 python-dateutil-2.9.0.post0 pytz-2024.2 six-1.16.0 terminaltables-3.1.10 typing-extensions-4.1.1 wcwidth-0.2.13 zipp-3.6.0 

===> Modify profile to update your $PATH and enable shell/tab completion now? (Y/n): n 

-- If you change your mind, add 'source /home/oracle/lib/oracle-cli/lib/python3.6/site-packages/oci_cli/bin/oci_autocomplete.sh' to your rc file and restart your shell to enable tab completion. 

-- You can run the CLI with '/home/oracle/bin/oci'. 

-- Installation successful. 

-- Run the CLI with /home/oracle/bin/oci --help 

Configure credenials to OCI user and copy API private key to host:

oracle@agent1:~/# vi .oci/config 
[DEFAULT] 
user=ocid1.user.ooracle@agent1:~/#
fingerprint=.*********************************
tenancy=ocid1.tenancy..*********************************
region=eu-zurich-1 
key_file=/home/oracle/.oci/oem_ro.pem 

 
#Create private key you can copy or create by copy/past with vi
oracle@agent1:~/# vi /home/oracle/.oci/oem_ro.pem 
 -----BEGIN PRIVATE KEY----- 
*** 
*** 
*** 
*** 
*** 
-----END PRIVATE KEY----- 

# Test your connection and gather some info for example compartment list
oracle@agent1:~/#  oci iam compartment list
{ 
  "data": [ 
    { 
      "compartment-id": "ocid1.tenancy.oc1..*********************************", 
      "defined-tags": { 
        "Oracle-Tags": { 
          "CreatedBy": "*********************************@*********************************.com", 
          "CreatedOn": "2023-06-15T15:28:03.408Z" 
        } 
      }, 
      "description": "Exadata Cloud@Customer database service", 
      "freeform-tags": {}, 
      "id": "ocid1.compartment.o*********************************", 
      "inactive-status": null, 
      "is-accessible": null, 
      "lifecycle-state": "ACTIVE", 
      "name": "ExaCC*********************************", 
      "time-created": "2023-06-15T15:28:03.545000+00:00" 
    } 
  ] 
} 

Set https proxy for your OEM agent (which later will be used to communicate with OCI)

edit emd.properties and add on the end 2 extra parameters.

oci_http_proxy_host=

oci_http_proxy_port=

#Configure proxy on your agent installation folder: 

oracle@agent1:~/#  vi/u01/app/oracle/agent/agent_inst/sysman/config/emd.properties
oci_http_proxy_host=proxy.proxy.com 
oci_http_proxy_port=3038 

#Restart Agent
oracle@agent1:~/# emctl stop agent 
Oracle Enterprise Manager Cloud Control 13c Release 5 
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved. 
Stopping agent ... stopped. 

oracle@agent1:~/#  [emagent1350] emctl start agent 
Oracle Enterprise Manager Cloud Control 13c Release 5 
Copyright (c) 1996, 2021 Oracle Corporation.  All rights reserved. 
Starting agent ........... started. 

Configure named credentials in OEM for OCI :

Go to Setup > Security > Named Credentials Click Create tab and create credentials using keys and OCI tenants keys copied from config or credentials:

next you can test it choice agent on which we installed oci and configure proxy.

Deploy Exadata plugins on Agents:

Go to Setup > Extensibility > Plug-ins

Choice:

Engineered Systems > Oracle Exadata

Click Deploy on and choice targets you wanna deploy it:

Follow logs to check if deployment successful :

Discovering ExaCC targets

Now we have all prerequisites configured and tested so we can discover our ExaCC :

Go to > Setup > Add Target > Add Target Manually > Add Using Guided Process

Choice target ExaCC infrastructure you wanna add to OEM , you can Also specify ExaCLI credentials to monitor Cellservers and ASMs (optional) We gather this credentials earlier in this tutorial.

Now you can Click next and Promote , and watch status of Target Promotions.

Please also turn on management packs for ExaCC

Go to Setup > Management Pack > Management Pack Access >

After Promotion finish you should have new entry in Target tab:

and Few minutes later you should also see your Oracle Cloud ExaCC infrastructure with all VMs and Databases.

You can go to each Exadata Cluster and see more details, also to ASM Grid Infrastructure:

Cell Statistics:

Please wait for some metrics and statistics some of them need up to 24h to be gathered.

One more interesting functionality of this integration is that there are new Dashboards

For Example Exadata Fleet Aggregated Capacity , Configuration , etc.

They can help you with overall look and capacity planning for future, you can see your total storage, memory current usage and historical values so you can predict growing trend in future.

I hope you enjoy this tutorial and already plan your ExaCC integration 😉

L’article ExaCC OCI integration with OEM 13.5 Cloud Control installed On-premises. est apparu en premier sur dbi Blog.

Oracle Database Appliance is a great piece of hardware: a vast majority of customers running on ODAs choose again ODAs when it’s time to renew their infrastructure. For sure, with Cloud strategies spreading around, on-premises solutions now have a serious challenger. But Cloud is not yet for all companies, and servers are still bought by customers for on-prem needs.

What’s special with ODA?

You may expect that ODA is something really special. You’re wrong. ODA’s hardware is not much different than what’s inside a classic server. And ODA is a sort of classic server because it’s nothing else than an x86_64 server from the Oracle Servers’ lineup. At the very beginning of the ODA history, ODAs were labelled “SunServer x86”, and you got a sticker in the box to put on the front panel with the correct “ODA X3-2” naming. So you may think that the price difference is for the sticker only, but actually it’s much more than that. What makes ODA so special is the dedicated software bundle. 95% of the software suite on the ODA is composed of the same pieces of software you would use on a normal Linux box, but it’s driven by a unique and dedicated CLI (odacli). Since X6-2, this CLI is also available as a GUI (or BUI) through your browser for a more user-friendly experience. With this CLI or this BUI, you will manage everything: the appliance provisioning, DB homes and databases creation/deletion, patching, managing the VMs, configuring the backups, aso. And you will do all these operations much faster than you would do on a classic infrastructure. The CLI from the first ODAs was very basic, the CLI from nowadays packs a lot of features and becomes more and more clever across the time. As this CLI is bundled with the patch, all customers and all ODAs will benefit from newest features after patching.

Hardware reliability

Reliability is something that has greatly improved these past years, on ODA and also globally. For sure, some components may fail when the server will get older, but it’s mostly disks, memory modules and PSUs: these components being redundant, your databases won’t stop in case of a failure. And as your ODA will probably be covered by the hardware support from the vendor, you will receive a replacement module as soon as a failure is predicted or detected. There is virtually no lifetime limit on an ODA as soon as the few components that may failed get replaced. Furthermore, reliability of SSDs seems better than reliability of HDDs. Despite having lower write cycles number, not having moving parts also means less wear as a result.

Software availability

When you look at the latest 19.18 patch for ODA, you will see that ODA X5-2 is still among the supported models. It means that you can still patch or do a fresh setup with 19c on top of this 8-year old server. This is something we’ve already tested at dbi services, as we have this kind of old ODA in our lab.

If you have a look at what happened before X5-2, ODA X4-2 was supported until 2020 and has been released in 2013. 7 years of software update is the average I’ve noticed working with ODA since years.

Disaster Recovery capabilities

Disaster Recovery is achieved through Data Guard for Enterprise Edition, and Dbvisit Standby for Standard Edition 2 (most of the time). ODA projects planned without considering a DR setup became very rare. It means that your databases are not tied with a single server anymore. Most of the critical databases can (and should) move to another server within minutes, meaning that if a datacenter or a serious hardware failure occurs, you just need to failover each database and everything is back to normal. Therefore, having a rather old server is not so critical nowadays. And this is the same for all kind of hardware.

Increasing cost of hardware

Oracle software technologies are not cheap as you know, and it’s the same for hardware. Furthermore, price of ODAs increased during these past months, like many other things. Value of an ODA is still great, because you will need less days to deploy and to maintain this kind of platform. But you may need to justify your investment even more than before. If you plan to keep these appliances 1 or 2 more year(s), it will probably ease making this investment.

Environmental considerations

Last but not least. Apart from customers coming from older disk-based ODAs (until X5-2 – most of the ODAs are now SSD-only), performance improvement across ODA generation is quite moderate. Since X6-2, performance is not the main reason for moving to new hardware. According to my experience, most common reasons are: reaching the planned lifetime according to customers’ standards, end of planned hardware support and no will to renew it, and disk capacity almost reached. Keeping your ODAs 7 years instead of 5 years simply means 30% less hardware wastage and can also justify a bigger disk capacity when ordering (defining accurate disk capacity requirements is quite tough for 5 years and beyond).

Conclusion

Please consider ODA as a long-lifetime platform. It definitely helps your company affording this kind of investment. And it also helps choosing a more comfortable disk setup and limiting your CO2 emission with lower pace hardware renewal.

L’article Why you should consider keeping your ODA more than 5 years est apparu en premier sur dbi Blog.

Patch 19.17 is now available on Oracle Database Appliance. It’s time to test it.

What’s new?

This version brings latest PSUs to database and grid homes with their bug fixes, as usual. It also brings latest 21.8 databases but as DB Systems only, 21c being an innovation release. There are not that many new features and it’s not bad. No major OS upgrade, no breaking new feature, no big changes. At some points, most of us prefer stability and easy-to-apply patches.

The only new features I’ve read from the release note are:

• Error correlation in the Browser User Interface (BUI), making a kind of crosscheck between odacli errors and log files from the various components. As you may know, odacli is just an interface on top of classic OS and Oracle tools, and it’s sometimes tough to find the log which refer to the error
• Database patching can now be done at the database level, and not only at the DB home level, with oracle update-database
• Enhanced patching when using Data Guard to limit downtime: I’m not so sure it will change something as it’s convenient to switch all the databases prior starting to patch

Which ODA is compatible with this 19.17?

The new ODAs X9-2S/L/HA and for sure X8, X7 and X6 series. X5-2HA is still on the compatibility list, so you can keep this old appliance up-to-date.

Is this patch a cumulative one?

This 19.17 can be applied on top of 19.13 or later. I just applied this new patch on an X7-2M running on 19.9 using 19.13 as an intermediate version.

Is there also a patch for my databases?

Only databases version 19c are now supported, and this is OK because this is the only one you should use now. Good bye 12.1. You may use Data Preserving Reprovisioning if you come from a very old version (<=18.8) and want to keep your 11g or 18c for example. I wrote a blog post on this feature a few months ago.

It’s not in the new feature list but it seems that it’s now possible to register an old DB clone without any problem. But I would recommend to only use 19c with this patch version. Using old databases is not supported, may not work properly with odacli, and you should be aware that 11g and 12cR1 are over now.

Download the patch and clone files

Download the patch and the corresponding clones to be able to apply the complete patch.

34753059 => the patch itself
30403673 => the GI clone needed for deploying newer GI version (mandatory)
30403662 => the DB clone for deploying new version of 19c

You don’t need the ISO file for patching, but I would recommend to download it (patch 30403643).

Be sure to choose the very latest 19.17 when downloading the clones, download link will first propose older versions.

Prepare the patching

Before running prepatch, please check these prerequisites:

• filesystems have 20% available free space (does not concern acfs volumes)
• additional rpms manually installed should be removed
• revert profile scripts to default’s one (for grid and oracle users)
• make sure you planned a generous downtime, 4 hours being the bare minimum for patching and troubleshooting. 1 day is never too much.

You should use odabr to make snapshots of the important filesystems prior patching. It’s fast and doesn’t cost anything. I would also recommend doing a backup of most important files: tnsnames.ora and listener.ora, database list, network configuration files, aso.

Version precheck

Start to check current version on all components:

odacli describe-component | grep -v ^$

System Version
---------------
19.13.0.0.0
System node Name
---------------
uns-oda2
Local System Version
---------------
19.13.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.13.0.0.0           up-to-date
GI                                        19.13.0.0.211019      up-to-date
DB {
[ OraDB19000_home3,OraDB19000_home4 ]     19.13.0.0.211019      up-to-date
}
DCSCONTROLLER                             19.13.0.0.0           up-to-date
DCSCLI                                    19.13.0.0.0           up-to-date
DCSAGENT                                  19.13.0.0.0           up-to-date
DCSADMIN                                  19.13.0.0.0           up-to-date
OS                                        7.9                   up-to-date
ILOM                                      5.0.2.24.r141466      up-to-date
BIOS                                      41100500              up-to-date
SHARED CONTROLLER FIRMWARE                QDV1RF32              up-to-date
LOCAL DISK FIRMWARE                       0121                  up-to-date
SHARED DISK FIRMWARE                      0121                  up-to-date
HMP                                       2.4.8.0.600           up-to-date

Once the patch will be registered in the ODA repository, the “Available Version” column will be updated with versions provided within the patch.

Prepararing the patch and updating the DCS tools

Copy the patch files on your ODA in a temp directory. On ODA X9-2 system disks are now smaller, so don’t hesitate to put the file on an nfs share or in the FRA acfs volume if you have acfs configured for your databases. Then unzip the files:

cd /u03/app/oracle
unzip -o p30403662_1917000_Linux-x86-64.zip  
unzip -o p30403673_1917000_Linux-x86-64.zip  
unzip -o p34753059_1917000_Linux-x86-64.zip

I first tried to register both patch and GI, but coming from 19.13 it doesn’t work:

odacli update-repository -f /u03/app/oracle/oda-sm-19.17.0.0.0-221126.1-server.zip,/u03/app/oracle/odacli-dcs-19.17.0.0.0-221029-GI-19.17.0.0.zip

DCS-10001:Internal error encountered: Cannot find the corresponding image for /u03/app/oracle/odacli-dcs-19.17.0.0.0-221029-GI-19.17.0.0.zip in img_metadata.

It will work if you come from a newer version, from 19.16 for example.

But it’s not a problem as you can first register the patch only, update the DCS components and then register the GI clone:

odacli update-repository -f /u03/app/oracle/oda-sm-19.17.0.0.0-221126.1-server.zip
odacli describe-job -i "dbacdf31-405f-4f08-acf2-21d78b66945a"

Job details
----------------------------------------------------------------
                     ID:  dbacdf31-405f-4f08-acf2-21d78b66945a
            Description:  Repository Update
                 Status:  Success
                Created:  December 2, 2022 9:55:39 AM CET
                Message:  /u03/app/oracle/oda-sm-19.17.0.0.0-221126.1-server.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle                             December 2, 2022 9:55:40 AM CET     December 2, 2022 9:56:09 AM CET     Success

odacli describe-component | grep -v ^$ 
System Version
---------------
19.13.0.0.0
System node Name
---------------
uns-oda2
Local System Version
---------------
19.13.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.13.0.0.0           19.17.0.0.0
GI                                        19.13.0.0.211019      19.17.0.0.221018
DB {
[ OraDB19000_home3,OraDB19000_home4 ]     19.13.0.0.211019      19.17.0.0.221018
}
DCSCONTROLLER                             19.13.0.0.0           19.17.0.0.0
DCSCLI                                    19.13.0.0.0           19.17.0.0.0
DCSAGENT                                  19.13.0.0.0           19.17.0.0.0
DCSADMIN                                  19.13.0.0.0           19.17.0.0.0
OS                                        7.9                   up-to-date
ILOM                                      5.0.2.24.r141466      5.1.0.23.r146986
BIOS                                      41100500              41120100
SHARED CONTROLLER FIRMWARE                QDV1RF32              QDV1RF35
LOCAL DISK FIRMWARE                       0121                  not-available
SHARED DISK FIRMWARE                      0121                  not-available
HMP                                       2.4.8.0.600           2.4.8.9.603

Patching from 19.13 will normally be easy as there are not that many changes.

odacli update-dcsadmin -v 19.17.0.0.0
sleep 60; odacli describe-job -i "1b7ddd74-6108-4480-8584-251932476f44"
Job details
----------------------------------------------------------------
                     ID:  1b7ddd74-6108-4480-8584-251932476f44
            Description:  DcsAdmin patching
                 Status:  Success
                Created:  December 2, 2022 9:57:42 AM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                December 2, 2022 9:57:42 AM CET     December 2, 2022 9:57:42 AM CET     Success
dcs-admin upgrade                        December 2, 2022 9:57:42 AM CET     December 2, 2022 9:57:49 AM CET     Success

 
odacli update-dcscomponents -v 19.17.0.0.0
This job is interactive
{
  "jobId" : "c3ed2039-82b4-45a4-b7d4-813da9b2aa1a",
  "status" : "Success",
  "message" : "Update-dcscomponents is successful on all the node(s):DCS-Agent shutdown is successful. MySQL upgrade is successful. Metadata migration is done before. Metadata schema update is done. dcsagent RPM upgrade is successful.  dcscli RPM upgrade is successful.  dcscontroller RPM upgrade is successful.  Successfully ran setupAgentAuth.sh zookeeper RPM upgrade is successful.  ",
  "reports" : null,
  "createTimestamp" : "December 02, 2022 10:03:37 AM CET",
  "description" : "Update-dcscomponents job completed and is not part of Agent job list",
  "updatedTime" : "December 02, 2022 10:05:16 AM CET"
}

odacli update-dcsagent -v 19.17.0.0.0
sleep 180; odacli describe-job -i "716a8259-1404-454e-90fb-cf59c14921d7"
Job details
----------------------------------------------------------------
                     ID:  716a8259-1404-454e-90fb-cf59c14921d7
            Description:  DcsAgent patching
                 Status:  Success
                Created:  December 2, 2022 10:06:21 AM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Stop DCS Admin                           December 2, 2022 10:06:22 AM CET    December 2, 2022 10:06:22 AM CET    Success
Generate mTLS certificates               December 2, 2022 10:06:22 AM CET    December 2, 2022 10:06:23 AM CET    Success
Exporting Public Keys                    December 2, 2022 10:06:23 AM CET    December 2, 2022 10:06:24 AM CET    Success
Creating Trust Store                     December 2, 2022 10:06:24 AM CET    December 2, 2022 10:06:26 AM CET    Success
Update config files                      December 2, 2022 10:06:26 AM CET    December 2, 2022 10:06:26 AM CET    Success
Restart DCS Admin                        December 2, 2022 10:06:26 AM CET    December 2, 2022 10:06:46 AM CET    Success
Dcs-agent upgrade  to version 19.17.0.0.0 December 2, 2022 10:06:46 AM CET    December 2, 2022 10:07:58 AM CET    Success
Update System version                    December 2, 2022 10:07:58 AM CET    December 2, 2022 10:07:58 AM CET    Success

odacli update-repository -f /u03/app/oracle/odacli-dcs-19.17.0.0.0-221029-GI-19.17.0.0.zip
sleep 60; odacli describe-job -i "4f81cf82-fed7-47a0-a6c3-69d3f9193e65"

Job details
----------------------------------------------------------------
                     ID:  4f81cf82-fed7-47a0-a6c3-69d3f9193e65
            Description:  Repository Update
                 Status:  Success
                Created:  December 2, 2022 10:09:44 AM CET
                Message:  /u03/app/oracle/odacli-dcs-19.17.0.0.0-221029-GI-19.17.0.0.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle                             December 2, 2022 10:09:45 AM CET    December 2, 2022 10:10:21 AM CET    Success

Prepatching report

Let’s do the prepatching test:

odacli create-prepatchreport -s -v 19.17.0.0.0
sleep 600 ; odacli describe-prepatchreport -i bb50b38f-55be-424a-bdf9-7d70206a61e1

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  bb50b38f-55be-424a-bdf9-7d70206a61e1
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER, SERVER]
                 Status:  FAILED
                Created:  December 2, 2022 10:12:55 AM CET
                 Result:  One or more pre-checks failed for [GI]

Node Name
---------------
uns-oda2

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
 
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Failed    Lock on central inventory detected:
                                          /u01/app/oraInventory/locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Validate space for clones       Success   Clones volume is already created
volume
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate command execution      Success   Validated command execution

On my configuration it didn’t work but this failure is a known issue:

rm -rf /u01/app/oraInventory/locks

odacli create-prepatchreport -s -v 19.17.0.0.0
sleep 600 ; odacli describe-prepatchreport -i d334c952-7bd8-4dc1-bf44-ee9247f7acc2

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  d334c952-7bd8-4dc1-bf44-ee9247f7acc2
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER, SERVER]
                 Status:  SUCCESS
                Created:  December 2, 2022 10:23:01 AM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
uns-oda2

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
 
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Validate space for clones       Success   Clones volume is already created
volume
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate command execution      Success   Validated command execution

Everything is OK to start patching.

Patching server and GI

Let’s start the update-server:

odacli update-server -v 19.17.0.0.0
odacli describe-job -i "6e32cf55-54b5-41fc-af07-0cb773e68132"
Job details
----------------------------------------------------------------
                     ID:  6e32cf55-54b5-41fc-af07-0cb773e68132
            Description:  Server Patching
                 Status:  Success
                Created:  December 2, 2022 10:35:33 AM CET
                Message:  Successfully patched GI with RHP

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validating GI user metadata              December 2, 2022 10:35:46 AM CET    December 2, 2022 10:35:46 AM CET    Success
Validate ILOM server reachable           December 2, 2022 10:35:46 AM CET    December 2, 2022 10:35:46 AM CET    Success
Validate DCS Admin mTLS setup            December 2, 2022 10:35:46 AM CET    December 2, 2022 10:35:46 AM CET    Success
Configure export clones resource         December 2, 2022 10:35:47 AM CET    December 2, 2022 10:35:48 AM CET    Success
Creating repositories using yum          December 2, 2022 10:35:48 AM CET    December 2, 2022 10:35:52 AM CET    Success
Updating YumPluginVersionLock rpm        December 2, 2022 10:35:52 AM CET    December 2, 2022 10:35:52 AM CET    Success
Applying OS Patches                      December 2, 2022 10:35:52 AM CET    December 2, 2022 10:46:25 AM CET    Success
Creating repositories using yum          December 2, 2022 10:46:26 AM CET    December 2, 2022 10:46:26 AM CET    Success
Applying HMP Patches                     December 2, 2022 10:46:26 AM CET    December 2, 2022 10:46:46 AM CET    Success
Patch location validation                December 2, 2022 10:46:46 AM CET    December 2, 2022 10:46:46 AM CET    Success
Oda-hw-mgmt upgrade                      December 2, 2022 10:46:47 AM CET    December 2, 2022 10:47:21 AM CET    Success
OSS Patching                             December 2, 2022 10:47:21 AM CET    December 2, 2022 10:47:21 AM CET    Success
Applying Firmware Disk Patches           December 2, 2022 10:47:21 AM CET    December 2, 2022 10:47:23 AM CET    Success
Applying Firmware Controller Patches     December 2, 2022 10:47:24 AM CET    December 2, 2022 10:48:12 AM CET    Success
Checking Ilom patch Version              December 2, 2022 10:48:12 AM CET    December 2, 2022 10:48:13 AM CET    Success
Patch location validation                December 2, 2022 10:48:13 AM CET    December 2, 2022 10:48:13 AM CET    Success
Save password in Wallet                  December 2, 2022 10:48:13 AM CET    December 2, 2022 10:48:13 AM CET    Success
Disabling IPMI v2                        December 2, 2022 10:48:13 AM CET    December 2, 2022 10:48:14 AM CET    Success
Apply Ilom patch                         December 2, 2022 10:48:14 AM CET    December 2, 2022 10:57:11 AM CET    Success
Copying Flash Bios to Temp location      December 2, 2022 10:57:11 AM CET    December 2, 2022 10:57:11 AM CET    Success
Starting the clusterware                 December 2, 2022 10:57:11 AM CET    December 2, 2022 10:58:54 AM CET    Success
Registering image                        December 2, 2022 10:58:54 AM CET    December 2, 2022 10:58:54 AM CET    Success
Registering working copy                 December 2, 2022 10:58:54 AM CET    December 2, 2022 10:58:54 AM CET    Success
Registering image                        December 2, 2022 10:58:54 AM CET    December 2, 2022 10:58:54 AM CET    Success
Creating GI home directories             December 2, 2022 10:58:54 AM CET    December 2, 2022 10:58:54 AM CET    Success
Extract GI clone                         December 2, 2022 10:58:54 AM CET    December 2, 2022 10:58:55 AM CET    Success
Provisioning Software Only GI with RHP   December 2, 2022 10:58:55 AM CET    December 2, 2022 10:58:55 AM CET    Success
Patch GI with RHP                        December 2, 2022 10:58:55 AM CET    December 2, 2022 11:05:10 AM CET    Success
Updating GIHome version                  December 2, 2022 11:05:10 AM CET    December 2, 2022 11:05:14 AM CET    Success
Validate GI availability                 December 2, 2022 11:05:52 AM CET    December 2, 2022 11:05:52 AM CET    Success
Patch KVM CRS type                       December 2, 2022 11:05:52 AM CET    December 2, 2022 11:05:52 AM CET    Success
Patch VM vDisks CRS dependencies         December 2, 2022 11:05:52 AM CET    December 2, 2022 11:05:52 AM CET    Success
Patch DB System domain config            December 2, 2022 11:05:53 AM CET    December 2, 2022 11:05:53 AM CET    Success
Update System version                    December 2, 2022 11:05:53 AM CET    December 2, 2022 11:05:53 AM CET    Success
Cleanup JRE Home                         December 2, 2022 11:05:53 AM CET    December 2, 2022 11:05:53 AM CET    Success
Add SYSNAME in Env                       December 2, 2022 11:05:53 AM CET    December 2, 2022 11:05:53 AM CET    Success
Starting the clusterware                 December 2, 2022 11:05:53 AM CET    December 2, 2022 11:05:53 AM CET    Success
Setting ACL for disk groups              December 2, 2022 11:05:53 AM CET    December 2, 2022 11:05:57 AM CET    Success
Enable LKCE                              December 2, 2022 11:07:50 AM CET    December 2, 2022 11:10:37 AM CET    Success
Update previous workarounds              December 2, 2022 11:10:37 AM CET    December 2, 2022 11:10:37 AM CET    Success
Generating and saving BOM                December 2, 2022 11:10:37 AM CET    December 2, 2022 11:12:32 AM CET    Success
PreRebootNode Actions                    December 2, 2022 11:12:32 AM CET    December 2, 2022 11:13:19 AM CET    Success
Reboot Ilom                              December 2, 2022 11:13:19 AM CET    December 2, 2022 11:13:19 AM CET    Success

Server reboots 5 minutes after the patch ends. On this X7-2M server patching lasted 40 minutes.

Checking filesystems told me that there is a problem on the /boot. It’s quite common on an ODA where multiple patches were applied. If you have this problem, please identify the old kernels and remove the rpms packages:

df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        474M  451M     0 100% /boot

uname -a
Linux uns-oda2 4.14.35-2047.518.4.1.el7uek.x86_64 #2 SMP Tue Oct 18 18:08:21 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux
rpm -qa | grep kernel-uek-4
kernel-uek-4.14.35-2047.508.3.2.el7uek.x86_64
kernel-uek-4.14.35-2025.400.9.el7uek.x86_64
kernel-uek-4.14.35-2047.518.4.1.el7uek.x86_64

rpm -e kernel-uek-4.14.35-2025.400.9.el7uek.x86_64

df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        474M  331M  119M  74% /boot

Patching the storage

Patching the storage is only needed if describe-component tells you that you’re not up-to-date. On my X7-2M I had to patch:

odacli update-storage -v 19.17.0.0.0
odacli describe-job -i "93199b6d-b400-43cf-812f-20ffaf59d704"
Job details
----------------------------------------------------------------
                     ID:  93199b6d-b400-43cf-812f-20ffaf59d704
            Description:  Storage Firmware Patching
                 Status:  Success
                Created:  December 2, 2022 11:29:47 AM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Applying Firmware Disk Patches           December 2, 2022 11:29:53 AM CET    December 2, 2022 11:29:55 AM CET    Success
Applying Firmware Controller Patches     December 2, 2022 11:29:56 AM CET    December 2, 2022 11:37:21 AM CET    Success
Generating and saving BOM                December 2, 2022 11:37:21 AM CET    December 2, 2022 11:38:28 AM CET    Success
PreRebootNode Actions                    December 2, 2022 11:38:28 AM CET    December 2, 2022 11:38:28 AM CET    Success
Reboot Ilom                              December 2, 2022 11:38:28 AM CET    December 2, 2022 11:38:28 AM CET    Success

I never encountered troubles during storage patching, so it should be fine.

Patching the DB homes

Time for patching the DB homes depends on the number of DB homes and number of databases. In this example, I will apply the patch on my two 19c DB homes:

odacli list-dbhomes
ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
f33e91e6-ac8c-48b1-a599-2e32ddf46b30     OraDB19000_home3     19.13.0.0.211019                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_3 CONFIGURED
804165be-d58e-44c6-b51d-c2729a99b8e9     OraDB19000_home4     19.13.0.0.211019                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4 CONFIGURED

The DB clone is needed:

cd /u03/app/oracle/
unzip -o p30403662_1917000_Linux-x86-64.zip
odacli update-repository -f /u03/app/oracle/odacli-dcs-19.17.0.0.0-221029-DB-19.17.0.0.zip
odacli describe-job -i "5ab4fcaa-2452-4f43-b329-cc3df03abbf7"

Job details
----------------------------------------------------------------
                     ID:  5ab4fcaa-2452-4f43-b329-cc3df03abbf7
            Description:  Repository Update
                 Status:  Success
                Created:  December 2, 2022 11:50:02 AM CET
                Message:  /u03/app/oracle/odacli-dcs-19.17.0.0.0-221029-DB-19.17.0.0.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle                             December 2, 2022 11:50:02 AM CET    December 2, 2022 11:50:38 AM CET    Success

A prepatching is also needed here:

odacli create-prepatchreport -d -i f33e91e6-ac8c-48b1-a599-2e32ddf46b30 -v 19.17.0.0.0
odacli describe-prepatchreport -i 4142cc7e-edbf-4aa2-9db1-cf5f2ea2ab94

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  4142cc7e-edbf-4aa2-9db1-cf5f2ea2ab94
            Description:  Patch pre-checks for [DB, ORACHKDB]: DbHome is OraDB19000_home3
                 Status:  FAILED
                Created:  December 2, 2022 11:55:08 AM CET
                 Result:  One or more pre-checks failed for [ORACHK]

Node Name
---------------
uns-oda2

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__DB__
Validate DB Home ID             Success   Validated DB Home ID:
                                          f33e91e6-ac8c-48b1-a599-2e32ddf46b30
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate minimum agent version  Success   Validated minimum agent version
Is GI upgraded                  Success   Validated GI is upgraded
Validate available space for    Success   Validated free space required under
db                                        /u01
Validate dbHomesOnACFS          Success   User has configured disk group for
configured                                Database homes on ACFS
Validate Oracle base            Success   Successfully validated Oracle Base
Is DB clone available           Success   Successfully validated clone file
                                          exists
Evaluate DBHome patching with   Success   Successfully validated updating
RHP                                       dbhome with RHP.  and local patching
                                          is possible
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Failed    ORAchk validation failed: .
Validate command execution      Success   Validated command execution
Check for parameter             Failed    AHF-3744: Database parameter
global_names                              GLOBAL_NAMES is not set to
                                          recommended value
Check for parameter             Failed    AHF-3744: Database parameter
global_names                              GLOBAL_NAMES is not set to
                                          recommended value
Check for parameter             Failed    AHF-3744: Database parameter
global_names                              GLOBAL_NAMES is not set to
                                          recommended value

I don’t care about Orachk recommendations on my databases because it has been set like that. I will apply the patch on this DB home with the force option:

odacli update-dbhome -i f33e91e6-ac8c-48b1-a599-2e32ddf46b30 -v 19.17.0.0.0 -f
sleep 600 ; odacli describe-job -i "0e4a424b-9892-46d9-90d8-ebb61737ef03"

Job details
----------------------------------------------------------------
                     ID:  0e4a424b-9892-46d9-90d8-ebb61737ef03
            Description:  DB Home Patching: Home Id is f33e91e6-ac8c-48b1-a599-2e32ddf46b30
                 Status:  Success
                Created:  December 2, 2022 12:11:31 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Creating wallet for DB Client            December 2, 2022 12:12:18 PM CET    December 2, 2022 12:12:18 PM CET    Success
Patch databases by RHP                   December 2, 2022 12:12:18 PM CET    December 2, 2022 12:18:34 PM CET    Success
Updating database metadata               December 2, 2022 12:18:34 PM CET    December 2, 2022 12:18:34 PM CET    Success
Set log_archive_dest for Database        December 2, 2022 12:18:34 PM CET    December 2, 2022 12:18:37 PM CET    Success
Patch databases by RHP                   December 2, 2022 12:18:37 PM CET    December 2, 2022 12:20:09 PM CET    Success
Updating database metadata               December 2, 2022 12:20:09 PM CET    December 2, 2022 12:20:09 PM CET    Success
Set log_archive_dest for Database        December 2, 2022 12:20:09 PM CET    December 2, 2022 12:20:13 PM CET    Success
Patch databases by RHP                   December 2, 2022 12:20:13 PM CET    December 2, 2022 12:25:58 PM CET    Success
Updating database metadata               December 2, 2022 12:25:58 PM CET    December 2, 2022 12:25:58 PM CET    Success
Set log_archive_dest for Database        December 2, 2022 12:25:58 PM CET    December 2, 2022 12:26:00 PM CET    Success
Update System version                    December 2, 2022 12:26:00 PM CET    December 2, 2022 12:26:00 PM CET    Success
Generating and saving BOM                December 2, 2022 12:26:00 PM CET    December 2, 2022 12:28:03 PM CET    Success
TDE parameter update                     December 2, 2022 12:28:39 PM CET    December 2, 2022 12:28:39 PM CET    Success

It’s the exact same procedure for the second DB home.

New DB homes have been created and my databases are now linked to these new ones:

odacli list-dbhomes
ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
f33e91e6-ac8c-48b1-a599-2e32ddf46b30     OraDB19000_home3     19.13.0.0.211019                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_3 CONFIGURED
804165be-d58e-44c6-b51d-c2729a99b8e9     OraDB19000_home4     19.13.0.0.211019                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4 CONFIGURED
7741bb27-43cc-484c-9525-5f854262c33d     OraDB19000_home5     19.17.0.0.221018                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5 CONFIGURED
cacbc994-04ae-4f40-ae96-a5b57bb3d89a     OraDB19000_home6     19.17.0.0.221018                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_6 CONFIGURED

odacli list-databases
ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
cbc31d02-5507-4e90-97b4-42a65bb2e2b7     PCGO_LCDF  SI       19.17.0.0.221018     false      OLTP     odb1     ACFS       CONFIGURED   7741bb27-43cc-484c-9525-5f854262c33d
118c79a4-c5cc-463c-ae29-d92208f502a7     PCSGO_LCDF SI       19.17.0.0.221018     false      OLTP     odb1     ACFS       CONFIGURED   cacbc994-04ae-4f40-ae96-a5b57bb3d89a
4892fc73-e89a-4fc4-8fac-c4d4f91ba64d     TCGO       SI       19.17.0.0.221018     false      OLTP     odb1     ACFS       CONFIGURED   7741bb27-43cc-484c-9525-5f854262c33d
ef0cc1fe-86b8-4f6d-9abe-a619147337ca     TCSGO      SI       19.17.0.0.221018     false      OLTP     odb1     ACFS       CONFIGURED   7741bb27-43cc-484c-9525-5f854262c33d

The old DB homes can now be safely removed:

odacli delete-dbhome -i f33e91e6-ac8c-48b1-a599-2e32ddf46b30
sleep 60 ;  odacli delete-dbhome -i 804165be-d58e-44c6-b51d-c2729a99b8e9
sleep 120 ; odacli list-jobs | tail -n 3
212c0662-a062-40c6-925b-3b915effa23d     Database Home OraDB19000_home3 Deletion with id f33e91e6-ac8c-48b1-a599-2e32ddf46b30 December 2, 2022 1:01:09 PM CET     Success
a64b32ac-9cad-4f23-acec-66077a5f856d     Database Home OraDB19000_home4 Deletion with id 804165be-d58e-44c6-b51d-c2729a99b8e9 December 2, 2022 1:01:41 PM CET     Success

Final checks

Let’s get the final versions:

odacli describe-component | grep -v ^$
System Version
---------------
19.17.0.0.0
System node Name
---------------
uns-oda2
Local System Version
---------------
19.17.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.17.0.0.0           up-to-date
GI
                                          19.17.0.0.221018      up-to-date
DB {
[OraDB19000_home5 [PCGO_LCDF,TCGO,
TCSGO]]                                   19.17.0.0.221018      up-to-date
[OraDB19000_home6 [PCSGO_LCDF]]
                                          19.17.0.0.221018      up-to-date
}
DCSCONTROLLER
                                          19.17.0.0.0           up-to-date
 
DCSCLI
                                          19.17.0.0.0           up-to-date
DCSAGENT
                                          19.17.0.0.0           up-to-date
DCSADMIN
                                          19.17.0.0.0           up-to-date
OS
                                          7.9                   up-to-date
ILOM
                                          5.1.0.23.r146986      up-to-date
BIOS
                                          41120100              up-to-date
LOCAL CONTROLLER FIRMWARE {
[c6]
                                          80000690              up-to-date
[c7]
                                          214.2.271.9           up-to-date
}
SHARED CONTROLLER FIRMWARE
                                          QDV1RF35              up-to-date
LOCAL DISK FIRMWARE
                                          N2010121              up-to-date
SHARED DISK FIRMWARE
                                          N2010121              up-to-date
HMP
                                          2.4.8.9.603           up-to-date

Everything is fine. I’m quiet for 2-3 months.

Cleanse the old patches

The old patches will never be used again, so don’t forget to remove them from the repository if your ODA has already been patched:

odacli cleanup-patchrepo -cl -v 19.13.0.0.0

odacli describe-job -i "7e1506b9-5778-47ed-84eb-65d976819c80"
Job details
----------------------------------------------------------------
                     ID:  7e1506b9-5778-47ed-84eb-65d976819c80
            Description:  Cleanup patchrepos
                 Status:  Success
                Created:  December 2, 2022 12:56:14 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Cleanup Repository                       December 2, 2022 12:56:14 PM CET    December 2, 2022 12:56:14 PM CET    Success
Cleanup JRE Home                         December 2, 2022 12:56:15 PM CET    December 2, 2022 12:56:15 PM CET    Success
Cleanup old ASR rpm                      December 2, 2022 12:56:15 PM CET    December 2, 2022 12:56:15 PM CET    Success

Put back your own settings

Once everything is OK, don’t forget to put back your settings:

• add your additional rpms manually if needed
• put back your profile scripts for grid and oracle users
• …

Patching DB Systems

If you use DB Systems on your ODA, meaning that some of your databases are running in dedicated VMs, you will need to apply the patch inside each DB System. I haven’t tried on my X7-2M because it only runs bare metal databases, but you will need to list your DB Systems, connect to each one, and apply the server and database patches:

odacli list-dbsystems
ssh ...
odacli update-server -v 19.17.0.0.0
odacli update-dbhome -i ... -v 19.17.0.0.0

Conclusion

This release is easy to apply coming from 19.13 and also probably from newer versions. Future releases may include a major OS upgrade and 23c sooner or later.

L’article ODA version 19.17 is available: how to patch? est apparu en premier sur dbi Blog.

Apart from the number of servers (1 vs 2), the main difference between Oracle Database Appliance lite (S/L) and High-Availability (HA) is the data disks location. They are inside the server on lite ODAs, and in a dedicated disk enclosure on HA ODAs. Obviously, this is because when 2 nodes want to use the same disks, these disks have to be shared. And this is why HA needs a SAS disk enclosure.

Disk technology on ODA

On lite ODAs, disks are SSDs inside the server and connected to the PCI Express bus without any interface, it’s the NVMe technology. This is very fast. There are faster technologies, like NVRam, but price/performance ratio made NVMe technology a game changer.

HA ODAs are not that fast regarding disk bandwidth. This is because NVMe only works for disks locally connected to the server’s motherboard. Both HA ODA nodes come with SAS controllers, these being connected to a SAS disk enclosure with SAS SSDs in it. As this enclosure is quite big (same height as the 2 nodes together), disk capacity is much higher than lite ODAs. A fully loaded X9-2HA ODA with SSDs has 184TB, it’s more than twice the 81TB capacity of a fully loaded ODA X9-2L. Furthermore, you can add another storage enclosure to X9-2HA to double the disk capacity to 369TB. And if you need even more capacity, there is an high capacity version of this enclosure with a mix of SSDs and HDDs for a maximum RAW capacity of 740TB. This is huge!

Hardware monitoring on ODA

Monitoring the ODA hardware is done from ILOM, the management console. ILOM can send SNMP traps and raise an alert if something is wrong. For an HA ODA, you have 2 ILOMs to monitor, as the 2 nodes are separate hardware. There’s a catch when it comes to monitoring the storage enclosure. This enclosure is not active, meaning that it doesn’t have any intelligence, and therefore cannot raise any alert. And ILOM from the nodes is not aware of hardware outside the nodes. You may think that it’s not really a problem because data disks are monitored by ASM. But this enclosure also has SAS interfaces to get connected with the nodes. And if one of these interfaces is down, you may not detect the problem.

The use case

My customer has multiple HA ODAs, and I was doing a sanity checks of these ODAs. Everything was fine until I did an orachk on an X6-2HA:

odaadmcli orachk
INFO: 2022-11-16 16:41:11: Running orachk under /usr/bin/orachk Searching for running databases . . . . .
........
List of running databases registered in OCR
1. XXX
3. YYY
4. ZZZ 
5. All of above
6. None of above
Select databases from list for checking best practices. For multiple databases, select 5 for All or comma separated number like 1,2 etc [1-6][5]. 6
RDBMS binaries found at /u01/app/oracle/product/19.0.0.0/dbhome_1 and ORACLE_HOME not set. Do you want to set ORACLE_HOME to "/u01/app/oracle/product/19.0.0.0/dbhome_1"?[y/n][y] y
...
FAIL => Several enclosure components controllers might be down
...

This is not something nice to see. My storage enclosure has a problem.

I will do another check with odaadmcli:

odaadmcli show enclosure

        NAME        SUBSYSTEM         STATUS      METRIC

        E0_FAN0     Cooling           OK          4910 rpm
        E0_FAN1     Cooling           OK          4530 rpm
        E0_FAN2     Cooling           OK          4920 rpm
        E0_FAN3     Cooling           OK          4570 rpm
        E0_IOM0     Encl_Electronics  OK          -
        E0_IOM1     Encl_Electronics  Not availab -
        E0_PSU0     Power_Supply      OK          -
        E0_PSU1     Power_Supply      OK          -
        E0_TEMP0    Amb_Temp          OK          23 C
        E0_TEMP1    Midplane_Temp     OK          23 C
        E0_TEMP2    PCM0_Inlet_Temp   OK          29 C
        E0_TEMP3    PCM0_Hotspot_Temp OK          26 C
        E0_TEMP4    PCM1_Inlet_Temp   OK          44 C
        E0_TEMP5    PCM1_Hotspot_Temp OK          28 C
        E0_TEMP6    IOM0_Temp         OK          22 C
        E0_TEMP7    IOM1_Temp         OK          28 C

Enclosure is not visible through one of the SAS controller. Maybe there is a failure, but the node is not able to say that there is a failure. It may be related to an unplugged SAS cable, as I found on MOS.

Let’s do a validate storage topology:

odacli validate-storagetopology
INFO    : ODA Topology Verification
INFO    : Running on Node0
INFO    : Check hardware type
SUCCESS : Type of hardware found : X6-2
INFO    : Check for Environment(Bare Metal or Virtual Machine)
SUCCESS : Type of environment found : Bare Metal
INFO    : Check number of Controllers
SUCCESS : Number of Internal RAID bus controllers found : 1
SUCCESS : Number of External SCSI controllers found : 2
INFO    : Check for Controllers correct PCIe slot address
SUCCESS : Internal RAID controller   : 23:00.0
SUCCESS : External LSI SAS controller 0 : 03:00.0
SUCCESS : External LSI SAS controller 1 : 13:00.0
INFO    : Check if JBOD powered on
SUCCESS : 0JBOD : Powered-on
INFO    : Check for correct number of EBODS(2 or 4)
FAILURE : Check for correct number of EBODS(2 or 4) : 1
ERROR   : 1 EBOD found on the system, which is less than 2 EBODS with 1 JBOD
INFO    : Above details can also be found in the log file=/opt/oracle/oak/log/srvxxx/storagetopology/StorageTopology-2022-11-16-17:21:43_34790_17083.log

EBOD stands for Expanded Bunch Of Disks, which is not very clear. But as disks are OK, this is probably related to cabling or controller in the enclosure.

Solution

My customer went to the datacenter and first checked the cabling, but it was fine. Opening an SR on My Oracle Support quickly solved the problem. A new controller was sent, it was swapped in the enclosure with the defect one without any downtime, and everything is fine then.

Conclusion

There is absolutely no problem with the HA storage enclosure not being smart. You don’t need a smart storage for this kind of server, as ODA is a “Simple. Reliable. Affordable” solution.

In this particular case, it’s hard to detect that the failure is a real one. But my customer was using a RAC setup with a failure in one of the redundant components, maybe since months. It’s definitely not satisfying. From time to time, manual and human checks are still needed!

L’article Warning: ODA HA disk enclosure is not smart! est apparu en premier sur dbi Blog.

The reasons why I like this product are mostly related to the following points (non-exhaustive list) :

• Ease of installation and configuration
• Ease of use
• Lightness
• Stability
• Continuous evolution (new features)
• Documentation quality
• Technical support efficiency

Despite all these qualities, it can happen that Dbvisit doesn’t work as it should and some troubleshooting is required.
In this post, I will describe a problem I encountered on the Dbvisit Standby (version 10.1) environment of one of our customers.

Issue

The following error message appeared while running the dbvctl command to transfer and the apply archive logs :

Dbvisit Standby process for preposs still running on odaprep01 (pid=53835).
See trace file 53835_dbvctl_preposs_202112081932.trc for more details.
Exceeded RUNNING_MAX_TIMES_TRIED=1 attempts.
(if Dbvisit Standby process is no longer running, then delete lock file /u01/app/dbvisit/standby/pid/dbvisit_preposs.pid)

This seems to indicate that the archive logs transfer process did not succeed properly and got stuck on the server.

Let’s have a look to the existing processes :

[oracle@odaprep01 ~]$ ps -ef | grep dbvisit
oracle   33833 14053  0 Dec08 ?        00:00:00 sh -c /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_3_12003.3.dbvisit.202112081923.sqlplus.dbv 2>/dev/null
oracle   33834 33833  0 Dec08 ?        00:00:00 /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_3_12003.3.dbvisit.202112081923.sqlplus.dbv
oracle   34813 34812  0 Dec08 ?        00:00:00 /bin/sh -c /u01/app/dbvisit/standby/dbvctl -d preposs >/tmp/dbvisit_apply_logs_preposs.log 2>&1
oracle   34814 34813  0 Dec08 ?        00:00:00 /u01/app/dbvisit/standby/dbvctl                                             -d preposs
oracle   35239 34814  0 Dec08 ?        00:00:00 sh -c /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/34814.0.dbvisit.202112081924.sqlplus.dbv 2>/dev/null
oracle   35240 35239  0 Dec08 ?        00:00:00 /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/34814.0.dbvisit.202112081924.sqlplus.dbv
oracle   36929     1  0 Sep17 ?        02:54:14 /u01/app/dbvisit/standby/dbvctl                                             -d preposs -D start
oracle   38142 14053  0 Dec08 ?        00:00:00 sh -c /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_4_1.4.dbvisit.202112082033.sqlplus.dbv 2>/dev/null
oracle   38143 38142  0 Dec08 ?        00:00:00 /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_4_1.4.dbvisit.202112082033.sqlplus.dbv
oracle   41184     1  0 Sep03 ?        01:17:16 /u01/app/dbvisit/dbvnet/dbvnet -d start
oracle   44524 14053  0 Dec08 ?        00:00:00 sh -c /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_5_1.5.dbvisit.202112082143.sqlplus.dbv 2>/dev/null
oracle   44525 44524  0 Dec08 ?        00:00:00 /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_5_1.5.dbvisit.202112082143.sqlplus.dbv
oracle   45780 14053  0 Dec08 ?        00:00:00 sh -c /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_6_1.6.dbvisit.202112082253.sqlplus.dbv 2>/dev/null
oracle   45781 45780  0 Dec08 ?        00:00:00 /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_6_1.6.dbvisit.202112082253.sqlplus.dbv
oracle   51450 14053  0 00:03 ?        00:00:00 sh -c /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_7_1.7.dbvisit.202112090003.sqlplus.dbv 2>/dev/null
oracle   51451 51450  0 00:03 ?        00:00:00 /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_7_1.7.dbvisit.202112090003.sqlplus.dbv
oracle   53234 14053  0 01:13 ?        00:00:00 sh -c /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_8_1.8.dbvisit.202112090113.sqlplus.dbv 2>/dev/null
oracle   53235 53234  0 01:13 ?        00:00:00 /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_8_1.8.dbvisit.202112090113.sqlplus.dbv
oracle   54442 14053  0 02:23 ?        00:00:00 sh -c /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_9_1.9.dbvisit.202112090223.sqlplus.dbv 2>/dev/null
oracle   54443 54442  0 02:23 ?        00:00:00 /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_9_1.9.dbvisit.202112090223.sqlplus.dbv
oracle   59799 14053  0 03:33 ?        00:00:00 sh -c /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_10_1.10.dbvisit.202112090333.sqlplus.dbv 2>/dev/null
oracle   59800 59799  0 03:33 ?        00:00:00 /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_10_1.10.dbvisit.202112090333.sqlplus.dbv
oracle   60840 14053  0 04:43 ?        00:00:00 sh -c /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_11_1.11.dbvisit.202112090443.sqlplus.dbv 2>/dev/null
oracle   60841 60840  0 04:43 ?        00:00:00 /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_11_1.11.dbvisit.202112090443.sqlplus.dbv
oracle   61931 14053  0 05:53 ?        00:00:00 sh -c /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_12_1.12.dbvisit.202112090553.sqlplus.dbv 2>/dev/null
oracle   61932 61931  0 05:53 ?        00:00:00 /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_12_1.12.dbvisit.202112090553.sqlplus.dbv
oracle   64979 14053  0 07:03 ?        00:00:00 sh -c /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_13_1.13.dbvisit.202112090703.sqlplus.dbv 2>/dev/null
oracle   64980 64979  0 07:03 ?        00:00:00 /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_13_1.13.dbvisit.202112090703.sqlplus.dbv
oracle   67322 14053  0 08:13 ?        00:00:00 sh -c /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_14_1.14.dbvisit.202112090813.sqlplus.dbv 2>/dev/null
oracle   67323 67322  0 08:13 ?        00:00:00 /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_14_1.14.dbvisit.202112090813.sqlplus.dbv
oracle   68513 14053  0 09:23 ?        00:00:00 sh -c /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_15_1.15.dbvisit.202112090923.sqlplus.dbv 2>/dev/null
oracle   68514 68513  0 09:23 ?        00:00:00 /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_15_1.15.dbvisit.202112090923.sqlplus.dbv
oracle   71517 14053  0 10:33 ?        00:00:00 sh -c /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_16_1.16.dbvisit.202112091033.sqlplus.dbv 2>/dev/null
oracle   71518 71517  0 10:33 ?        00:00:00 /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_16_1.16.dbvisit.202112091033.sqlplus.dbv
oracle   72932 14053  0 11:43 ?        00:00:00 sh -c /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_17_1.17.dbvisit.202112091143.sqlplus.dbv 2>/dev/null
oracle   72933 72932  0 11:43 ?        00:00:00 /usr/sbin/fuser /u01/app/dbvisit/standby/tmp/14053_17_1.17.dbvisit.202112091143.sqlplus.dbv
oracle   78878     1  0 Sep03 ?        00:09:45 /u01/app/dbvisit/dbvagent/dbvagent -d start
[oracle@odaprep02 ~]$

Mmmh, quite a big mess here, isn’t ?

Troubleshooting

You certainly noticed that there is a lot of blocked processes running the fuser command. Actually, before transferring an archive logs from the primary server to the standby one, fuser is executed to ensure that the concerned archive log is not being used by another process (e.g RMAN backup). This behavior can be confirmed by analyzing the trace file (<dbvisit_home>/traces directory) generated by the dbvctl command :

209 11:08:41 main::UTIL_UNIX_is_file_open: run command: /usr/sbin/fuser /u03/app/oracle/fast_recovery_area/PRODOSS_DC41/archivelog/2021_12_09/o1_mf_1_162040_jv3oc7xb_.arc
20211209 11:08:41 main::UTIL_run_command: ORACLE_HOME: /u01/app/oracle/product/19.0.0.0/dbhome_2
20211209 11:08:41 main::UTIL_run_command: PATH: /usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/usr/sbin:/sbin:/u01/app/oracle/product/19.0.0.0/dbhome_2/bin
20211209 11:08:41 main::UTIL_run_command: LD_LIBRARY_PATH: /u01/app/dbvisit/standby/lib:/u01/app/oracle/product/19.0.0.0/dbhome_2/lib

The first action I took was to disable the Dbvisit jobs from the crontab – on both sides – to avoid some new processes to be generated.
Then I killed all the blocked processes listed above to start from a clean situation, and I deleted the PID file stored in the <dbvisit_home>/pid > directory.
Finally I executed manually the fuser command against an archive log and there I could see that the command remained stuck, without any result to return. Ctrl-C was needed to stop it. This strange behavior was also the same with some other commands like df  or lsof.

Root cause

By analyzing the system logs (/var/log/messages), I could observe that a NFS share mounted on the server was no longer reachable :

Dec  8 19:38:08 odaprep01 kernel: nfs: server 10.84.48.100 not responding, timed out
Dec  8 19:41:08 odaprep01 kernel: nfs: server 10.84.48.100 not responding, timed out
Dec  8 19:41:14 odaprep01 kernel: nfs: server 10.84.48.100 not responding, timed out

And that’s why the fuser command triggered by dbvctl never ended.

Now, you are probably wondering “the archive logs are not stored on the NFS, so why did this have an impact ?“.
Let me explain…
fuser is designed to access all processes at one time to determine if any of their files is stored on the local file system. To discover that, a stat() call is used to identify the attributes of the processes’ executable. If the executable is stored on a NFS, the stat() call result depends on the NFS availability to be successful and can hang if it’s not reachable.

To solve the issue, I had of course to unmount the unreachable NFS mount point (umount -l <mount_point>).
As Dbvisit were not working properly since several hours, the standby database was out of sync because of archive logs gap. Unfortunately, the missing archive logs where not present anymore into the Fast Recovery Area, so I had to restore them from the RMAN backup :

RMAN> restore archivelog from logseq=26160 until logseq=26190;

Starting restore at 09-DEC-2021 12:26:39
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=365 device type=DISK

channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=26169
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=26170
channel ORA_DISK_1: reading from backup piece /u03/app/oracle/fast_recovery_area/PREPOSS_DC41/backupset/2021_12_09/o1_mf_annnn_TAG20211209T002732_jv2hv54g_.bkp
channel ORA_DISK_1: piece handle=/u03/app/oracle/fast_recovery_area/PREPOSS_DC41/backupset/2021_12_09/o1_mf_annnn_TAG20211209T002732_jv2hv54g_.bkp tag=TAG20211209T002732
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=26160
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=26161
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=26162
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=26163
...
...
...
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=26188
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=26189
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=26190
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/admin/preposs/backup/20211209_111002_arc_PREPOSS_533362994_s2246_p1.bck
channel ORA_DISK_1: piece handle=/u01/app/oracle/admin/preposs/backup/20211209_111002_arc_PREPOSS_533362994_s2246_p1.bck tag=ARC_20211209_111002
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07
Finished restore at 09-DEC-2021 12:27:17

RMAN>

And finally, I was able to restart the dbvctl command to resolve the gap by transferring and applying the restored archive logs :

oracle@odaprep01:/u01/app/dbvisit/standby/trace/ [preposs] dbvctl -d preposs
=============================================================
Dbvisit Standby Database Technology (10.1.0_0_gba3a9e08) (pid 15459)
dbvctl started on odaprep01: Thu Dec  9 12:28:33 2021
=============================================================

>>> Obtaining information from standby database (RUN_INSPECT=Y)... done
    Thread: 1 Archive log gap: 35. Transfer log gap: 35
>>> Transferring Log file(s) from preposs on odaprep01 to odaprep02:

    thread 1 sequence 26160 (o1_mf_1_26160_jv3t012k_.arc)... done
    thread 1 sequence 26161 (o1_mf_1_26161_jv3szzng_.arc)... done
    thread 1 sequence 26162 (o1_mf_1_26162_jv3t00lo_.arc)... done
    thread 1 sequence 26163 (o1_mf_1_26163_jv3szzpx_.arc)... done
    ...
    ...
    ...
    thread 1 sequence 26188 (o1_mf_1_26188_jv3t0jgx_.arc)... done
    thread 1 sequence 26189 (o1_mf_1_26189_jv3t0jpb_.arc)... done
    thread 1 sequence 26190 (o1_mf_1_26190_jv3t0gr6_.arc)... done

>>> Dbvisit Archive Management Module (AMM)

    Config: number of archives to keep      = 0
    Config: number of days to keep archives = 1
    Config: archive backup count            = 1
    Config: diskspace full threshold        = 80%
==========

    Total number of archive logs   : 35
    Current disk percent full (/u03/app/oracle/fast_recovery_area/) = 10%
==========

    Current disk percent full (FRA) = 0%
==========

    Number of archive logs deleted = 0

=============================================================
dbvctl ended on odaprep01: Thu Dec  9 12:30:19 2021
=============================================================

Conclusion

That was not a Dbvisit issue. As I said at the very beginning, Dbvisit is a great tool .
Hint: if you want to get the fuser, df, lsof commands working even when a mounted NFS is temporarily unreachable, a solution would be to mount it with the soft option :

mount -o rw,soft host.server.com/share /mymountpoint

According to the documentation :
“soft
      Generates a soft mount of the NFS file system. If an error occurs, the stat() function returns with an error.
      If the option hard is used, stat() does not return until the file system is available.”

Hope this helps.

L’article When unreachable NFS share mess up your Dbvisit Standby configuration est apparu en premier sur dbi Blog.

While giving an ODA workshop, I thought it was a good opportunity to test patching to the new 19.12 release.
However during the update-server process we got a failure on the our ODA update to 19.12 failing on GI patch using RHP.

Let’s see how we fixed it…

So this is our starting point:

[root@dbi-oda-x8 tmp]# odacli describe-job -i d3efbbf5-f1db-4c5e-8f03-e8876f85a341

Job details
----------------------------------------------------------------
                     ID:  d3efbbf5-f1db-4c5e-8f03-e8876f85a341
            Description:  Server Patching
                 Status:  Failure
                Created:  November 4, 2021 11:01:15 AM CET
                Message:  DCS-10001:Internal error encountered: Fail to patch GI with RHP : DCS-10001:Internal error encountered: DCS-10001:Internal error encountered: clonemetadata.xml file is not present in local repository.
Update repository with new clones and retry same command...

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Server patching                          November 4, 2021 11:01:23 AM CET    November 4, 2021 11:22:16 AM CET    Failure
Server patching                          November 4, 2021 11:01:24 AM CET    November 4, 2021 11:22:16 AM CET    Failure
Creating repositories using yum          November 4, 2021 11:02:36 AM CET    November 4, 2021 11:02:38 AM CET    Success
Updating YumPluginVersionLock rpm        November 4, 2021 11:02:38 AM CET    November 4, 2021 11:02:38 AM CET    Success
Applying OS Patches                      November 4, 2021 11:02:38 AM CET    November 4, 2021 11:10:49 AM CET    Success
Creating repositories using yum          November 4, 2021 11:10:50 AM CET    November 4, 2021 11:10:50 AM CET    Success
Applying HMP Patches                     November 4, 2021 11:10:50 AM CET    November 4, 2021 11:11:08 AM CET    Success
Patch location validation                November 4, 2021 11:11:08 AM CET    November 4, 2021 11:11:08 AM CET    Success
oda-hw-mgmt upgrade                      November 4, 2021 11:11:09 AM CET    November 4, 2021 11:11:41 AM CET    Success
OSS Patching                             November 4, 2021 11:11:42 AM CET    November 4, 2021 11:11:42 AM CET    Success
Applying Firmware Disk Patches           November 4, 2021 11:11:42 AM CET    November 4, 2021 11:11:46 AM CET    Success
Applying Firmware Controller Patches     November 4, 2021 11:11:46 AM CET    November 4, 2021 11:11:50 AM CET    Success
Checking Ilom patch Version              November 4, 2021 11:11:50 AM CET    November 4, 2021 11:11:50 AM CET    Success
Patch location validation                November 4, 2021 11:11:50 AM CET    November 4, 2021 11:11:50 AM CET    Success
Save password in Wallet                  November 4, 2021 11:11:50 AM CET    November 4, 2021 11:11:51 AM CET    Success
Apply Ilom patch                         November 4, 2021 11:11:51 AM CET    November 4, 2021 11:20:48 AM CET    Success
Copying Flash Bios to Temp location      November 4, 2021 11:20:48 AM CET    November 4, 2021 11:20:48 AM CET    Success
Server patching                          November 4, 2021 11:20:48 AM CET    November 4, 2021 11:22:16 AM CET    Failure
Starting the clusterware                 November 4, 2021 11:20:48 AM CET    November 4, 2021 11:22:16 AM CET    Success
registering image                        November 4, 2021 11:22:16 AM CET    November 4, 2021 11:22:16 AM CET    Success
registering working copy                 November 4, 2021 11:22:16 AM CET    November 4, 2021 11:22:16 AM CET    Success
registering image                        November 4, 2021 11:22:16 AM CET    November 4, 2021 11:22:16 AM CET    Success
Creating GI home directories             November 4, 2021 11:22:16 AM CET    November 4, 2021 11:22:16 AM CET    Success
Extract GI clone                         November 4, 2021 11:22:16 AM CET    November 4, 2021 11:22:16 AM CET    Success
Provisioning Software Only GI with RHP   November 4, 2021 11:22:16 AM CET    November 4, 2021 11:22:16 AM CET    Success
Patch GI with RHP                        November 4, 2021 11:22:16 AM CET    November 4, 2021 11:22:16 AM CET    Failure

So basically we had a failure because of

Patch GI with RHP November 4, 2021 11:22:16 AM CET November 4, 2021 11:22:16 AM CET Failure

with as explanation the following message:

DCS-10001:Internal error encountered: Fail to patch GI with RHP : DCS-10001:Internal error encountered:

DCS-10001:Internal error encountered: clonemetadata.xml file is not present in local repository.

This basically sounded like with we were missing the GI clone for the the 19.12 in our local repository (/opt/oracle/oak/pkgrepos)

The first check we was to verify one more time that our update repository job, to import the GI clone, worked properly

[root@dbi-oda-x8 clones]# odacli describe-job -i 3792c33b-3804-44f7-9806-eddf21bb4939

Job details
----------------------------------------------------------------
                     ID:  3792c33b-3804-44f7-9806-eddf21bb4939
            Description:  Repository Update
                 Status:  Success
                Created:  November 4, 2021 10:20:26 AM CET
                Message:  /tmp/ODA/odacli-dcs-19.12.0.0.0-210822.1-DB-19.12.0.0.zip,/tmp/ODA/odacli-dcs-19.12.0.0.0-210822.1-GI-19.12.0.0.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle                             November 4, 2021 10:20:29 AM CET    November 4, 2021 10:21:56 AM CET    Success
registering image                        November 4, 2021 10:21:57 AM CET    November 4, 2021 10:21:57 AM CET    Success
registering image                        November 4, 2021 10:21:57 AM CET    November 4, 2021 10:21:57 AM CET    Success

OK as this was fine, we decided to check if the file is really physically there:

[root@dbi-oda-x8 ~]# cd /opt/oracle/oak/pkgrepos/orapkgs/clones/
[root@dbi-oda-x8 clones]# ls -l
total 22095968
-rw-r--r-- 1 root root      13813 Aug 23 19:06 clonemetadata.xml
-rw-r--r-- 1 root root 4500047202 Feb  9  2021 db19.210119.tar.gz
-r-xr-xr-x 1 root root 5033359682 Aug 23 19:10 db19.210720.tar.gz
-r-xr-xr-x 1 root root 6542844964 Aug 23 19:06 grid19.210720.tar.gz
-rw-r--r-- 1 root root 6542464864 Feb  9  2021 grid19.tar.gz
drwx------ 2 root root      65536 Nov  4 11:01 lost+found

Hmmm the file is there and with a simple grep command we can confirm that it is available in the clonemedata.xml too.

We started searching in the ODA 19.12 Release Note for the known issues and the end of the list we found the following one:

Ok it is not exactly the same message (not talking about clonemedata.xml) but still linked to missing version.
The proposed workaround is:

So let’s have a look to our setup on the source GI Home

[root@dbi-oda-x8 bin]# ls -l osdbagrp*
-rwxr-xr-x 1 root oinstall 33488 Nov  3 13:24 osdbagrp
-rw-r----- 1 root oinstall     0 Feb  9  2021 osdbagrp0

As in the know issue, the file is not belonging to grid user. We tried then to change the ownership

[root@dbi-oda-x8 bin]# chown grid osdbagrp


[root@dbi-oda-x8 bin]# ls -l osdbagrp*
-rwxr-xr-x 1 grid oinstall 33488 Nov  3 13:24 osdbagrp
-rw-r----- 1 root oinstall     0 Feb  9  2021 osdbagrp0

Then we run the command recommended to update the registry…unfortunately neither worked

[root@dbi-oda-x8 bin]# odacli update-registry -n gihome
DCS-10112:Specified components are already discovered.


[root@dbi-oda-x8 bin]# odacli update-registry -n system
DCS-10112:Specified components are already discovered.

As it didn’t wanted to update the information, we decided to force the update

[root@dbi-oda-x8 bin]# odacli update-registry -n gihome -f

Job details
----------------------------------------------------------------
                     ID:  4a9c149b-d59a-4a05-baea-2acfa446ec34
            Description:  Discover System Components : gihome
                 Status:  Created
                Created:  November 4, 2021 11:42:05 AM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

Checking the state of the job shows that it is sucessful

[root@dbi-oda-x8 bin]# odacli describe-job -i 4a9c149b-d59a-4a05-baea-2acfa446ec34

Job details
----------------------------------------------------------------
                     ID:  4a9c149b-d59a-4a05-baea-2acfa446ec34
            Description:  Discover System Components : gihome
                 Status:  Success
                Created:  November 4, 2021 11:42:05 AM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Rediscover GiHome                        November 4, 2021 11:42:05 AM CET    November 4, 2021 11:42:07 AM CET    Success

The final steps is then to re-run the update server command:

[root@dbi-oda-x8 bin]# odacli update-server -v 19.12.0.0.0
{
  "jobId" : "946922b3-aefa-4c75-b2e1-019e9d404118",
  "status" : "Created",
  "message" : "Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.",
  "reports" : [ ],
  "createTimestamp" : "November 04, 2021 11:42:57 AM CET",
  "resourceList" : [ ],
  "description" : "Server Patching",
  "updatedTime" : "November 04, 2021 11:42:57 AM CET"
}

After few minutes the patching of the server is finally successful

[root@dbi-oda-x8 ~]# odacli describe-job -i 946922b3-aefa-4c75-b2e1-019e9d404118

Job details
----------------------------------------------------------------
                     ID:  946922b3-aefa-4c75-b2e1-019e9d404118
            Description:  Server Patching
                 Status:  Success
                Created:  November 4, 2021 11:42:57 AM CET
                Message:  Successfully patched GI with RHP

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Creating repositories using yum          November 4, 2021 11:43:11 AM CET    November 4, 2021 11:43:13 AM CET    Success
Updating YumPluginVersionLock rpm        November 4, 2021 11:43:13 AM CET    November 4, 2021 11:43:13 AM CET    Success
Applying OS Patches                      November 4, 2021 11:43:13 AM CET    November 4, 2021 11:43:14 AM CET    Success
Creating repositories using yum          November 4, 2021 11:43:15 AM CET    November 4, 2021 11:43:15 AM CET    Success
Applying HMP Patches                     November 4, 2021 11:43:15 AM CET    November 4, 2021 11:43:16 AM CET    Success
Patch location validation                November 4, 2021 11:43:16 AM CET    November 4, 2021 11:43:16 AM CET    Success
oda-hw-mgmt upgrade                      November 4, 2021 11:43:16 AM CET    November 4, 2021 11:43:16 AM CET    Success
OSS Patching                             November 4, 2021 11:43:16 AM CET    November 4, 2021 11:43:16 AM CET    Success
Applying Firmware Disk Patches           November 4, 2021 11:43:16 AM CET    November 4, 2021 11:43:20 AM CET    Success
Applying Firmware Controller Patches     November 4, 2021 11:43:20 AM CET    November 4, 2021 11:43:24 AM CET    Success
Checking Ilom patch Version              November 4, 2021 11:43:24 AM CET    November 4, 2021 11:43:24 AM CET    Success
Patch location validation                November 4, 2021 11:43:24 AM CET    November 4, 2021 11:43:24 AM CET    Success
Save password in Wallet                  November 4, 2021 11:43:25 AM CET    November 4, 2021 11:43:25 AM CET    Success
Apply Ilom patch                         November 4, 2021 11:43:25 AM CET    November 4, 2021 11:43:25 AM CET    Success
Copying Flash Bios to Temp location      November 4, 2021 11:43:25 AM CET    November 4, 2021 11:43:25 AM CET    Success
Starting the clusterware                 November 4, 2021 11:43:25 AM CET    November 4, 2021 11:43:25 AM CET    Success
registering image                        November 4, 2021 11:43:25 AM CET    November 4, 2021 11:43:26 AM CET    Success
registering working copy                 November 4, 2021 11:43:26 AM CET    November 4, 2021 11:43:26 AM CET    Success
registering image                        November 4, 2021 11:43:26 AM CET    November 4, 2021 11:43:26 AM CET    Success
Creating GI home directories             November 4, 2021 11:43:26 AM CET    November 4, 2021 11:43:26 AM CET    Success
Extract GI clone                         November 4, 2021 11:43:26 AM CET    November 4, 2021 11:43:26 AM CET    Success
Provisioning Software Only GI with RHP   November 4, 2021 11:43:26 AM CET    November 4, 2021 11:43:26 AM CET    Success
Patch GI with RHP                        November 4, 2021 11:43:26 AM CET    November 4, 2021 11:50:05 AM CET    Success
Updating GIHome version                  November 4, 2021 11:50:06 AM CET    November 4, 2021 11:50:08 AM CET    Success
Update System version                    November 4, 2021 11:50:23 AM CET    November 4, 2021 11:50:23 AM CET    Success
Cleanup JRE Home                         November 4, 2021 11:50:23 AM CET    November 4, 2021 11:50:23 AM CET    Success
Add SYSNAME in Env                       November 4, 2021 11:50:23 AM CET    November 4, 2021 11:50:23 AM CET    Success
Setting ACL for disk groups              November 4, 2021 11:50:23 AM CET    November 4, 2021 11:50:27 AM CET    Success
preRebootNode Actions                    November 4, 2021 11:52:14 AM CET    November 4, 2021 11:55:06 AM CET    Success
Reboot Ilom                              November 4, 2021 11:55:06 AM CET    November 4, 2021 11:55:06 AM CET    Success

Here we go our ODA is now up-to-date in version 19.12, we can continue with:

• cleaning the repo
• manually cleaning the former GI Home
• Creating a Database Home Store
• …
• …

Enjoy!

L’article ODA update-server to 19.12 fails on Patch GI with RHP est apparu en premier sur dbi Blog.

Combining hard- and software and importance of documentation

Oracle Exadata is undoubtedly an interesting platform to run Oracle RDBMS on. It’s the combination of hard- and software that makes the difference to other platforms. Maybe you remember the famous release of Apple iPhone that redefined smartphone market using sweet spots on hard- and software?
For the readers technically interested, I recommend to have a look at documentation. See here for example on hardware details allowing to compare old and new. Please also find a comparison in the next section.
Regarding ZDLRA, one can find site planning, network configuration, hardware and software installation, and maintenance in documentation. To date, there is no documentation available on new features, but you may check all books at a later time.

What changed? A comparison

Please find in documentation hardware details on a single webpage. In whitepapers, Oracle briefly introduces features and list some benchmark results:

• X8M-2 whitepaper
• X8M-8 whitepaper
• X9M-2 whitepaper
• X9M-8 whitepaper

Compute nodes hardware

CPU scores are taken from geekbench 5 results database, take with caution, may not be reliable. No increase on X?M-8 compute nodes, as same hardware on both versions. Main reason to use X?M-8 is more max memory available per compute node (think of in-memory database option). No average latency given on IOPS, therefore results are not completely documented and one need to interpret them with caution.
Please note benefit using persistent memory on compute node (in memory mode, so to increase total amount of RAM, but no persistence) are not as big as using PMEM on storage node (app-direct mode, persistence). See here the modes explained.

Storage nodes hardware

No increase on flash storage and persistent memory capacity. PMEM bandwidth increase according to Intel. No average latency given on IOPS, therefore results are not completely documented and one need to interpret them with caution.
(1) Comparison on base frequency must be taken with caution as there are more factors on a CPU to influence overall system performance. Other factors are CPU quantity, max turbo frequency, cache sizes, number of Ultra Path Interconnects (UPI), thermal design power (TDP), PCIe version and root complex, number of memory channels, etc. (list not complete).
(2) Xeon 8352Y was set to use only 16 out of 32 available cores using Intel Speed Select Technology – Performance Profile. This technology may not be used to lower RDBMS license costs. On Exadata, use Capacity on Demand feature instead. Scale up possible, scale down not. Similar to Oracle Database Appliance. One may also use hard partitioning.

Full rack configuration

Performance increase between X8M and X9M platform according to Oracle tests (we don’t know how Oracle is testing and if test are consistent) are quite impressive. Interestingly, x9M full rack with high capacity storage uses 14 instead of 12 storage nodes. Besides taht, there ist more CPU power as more modern hardware available both on compute and storage nodes. No average latency given on IOPS, therefore results are not completely documented and one need to interpret them with caution.

Please note there are also other configurations available:

Oracle Exadata Configuration Assistant (OECA) for Exadata simplifies the elastic configuration process. OECA helps you to investigate and plan a variety of elastic configuration scenarios. Yet not updated with X9M platform data, but certainly in future. Download it here, an Oracle account is needed.

Please be advised: Performance increase is not only due to hardware change (latest Intel Xeon Ice lake CPUs, PCIe 4.0 instead 3.0, series 200 persistent memory instead 100, … you name it), but also to software which may be used on both X8 and X9M platforms. It’s interesting to see what implications there are when choosing a processor platform. Maybe to see AMD and/or even a port to ARM CPUs in future on Exadata?
Furthermore, I believe hardware tuning is the easiest of all methods to tune Oracle RDBMS. But not the one with biggest effects. There can be a bigger impact by tuning on SQL level or simply keep active data sets in databases small (e.g. one uses partitioning and offload data on a regular basis). Both SQL tuning and data archiving means that a tuning specialist needs to understand how end users use a service. And how the service is meant to be used. Unfortunately there is a difference between the two… Therefore tuning is individual in every company even using the exact same software. Writing this I got remembered to the 2009 german movie “Same Same But Different“…
To sum it up: Tuning is hard work, a true craft, sometimes even art… It’s advisable to tune with a scientific approach (knowing the effects before implementing them) and end tuning efforts when impact on performance from an end user perspective is too low. So get end users in the tuning boat.

What can be learned from cloud

Speaking on performance tuning, one can think of services that can not be run on public clouds because of network latency. That can be a reason to have an Exadata on customer side (Exadata Cloud@Customer offering). But one can learn from cloud services to engineer environments on-premises. I see at least two interesting aspects:

1. Automatisation
   Saves time and ensures quality. DBAs end up becoming developers. Running services autonomously by data mine data ditctionary (or simple use “statistics”, but this term is not as modern as “data mining”) is also part of automatisation.
2. Standardisation
   Many database workloads can fit in a few standards. Maybe you start to set them like T-shirt sizes S, M, L, XL on your oracle RDBMS offerings? No rule without exception, there are always and will be services demanding more or less. But don’t underestimate the power of Standardisation.
   T-shirt sizes are usefull to define limits when consolidate a large number of database on same hardware. Exadata offers Ressource Manager to limit CPU, memory and disk usage.

To me it’s important to understand that both automatisation and standardisation can be achieved running services on premises as well. But probably at higher costs and time efforts. Data can be the second valuable asset in a company after employees and must be protected (see section “On trust”), so money is not important. Time and agility are some main reasons why to move workload to cloud. But a supplier should not take decisions for the customer, it’s always him to decide.

On trust

At the end, trust between customer and supplier is an important factor to choose a solution. Trust is a cultural thing, not everybody act the same. This makes live quite interesting, don’t you think? Regarding Exadata Cloud@customer service, customers trust Oracle corporation to run their workloads and deal with their data even hardware stays in customers data center. Not every supplier is capable protecting data as many incidents showed in the past, and risks will increase with standardisation (less variants to choose from) and monopolisation (fewer vendors). There would be an interesting experiment on Cloud@customer services: What happens if one caps internet access to the Exadata hardware? Will databases continue to run? Yes, according to Thomas Teske (see comments below), but not suggested.

On appliances

A vendor may know it’s solution best on capabilities and limits, so it’s a logic step to also offer appliances. The more complex a solution, the more this can become true. Oracle RDBMS is a complex piece of software. Appliances and services have to fit in an existing environment. My experience is: The bigger the company, the bigger the environment, the more vendors/silos there are, the less standards used. Quite an impressive amount of people are dealing to migrate data and services from one platform to another. This is by far not every time successful. Furthermore, legacy platforms stay longer as expected. It’s easy to end up as a zoo keeper.

Time to refactor?

On the other hand: if only the vendor is able to build meaningful appliances to its software, did the vendor reach or even miss the point in time to refactor its software products? Oracle RDBMS was first commercially available in 1979. Honestly, I have no idea how much code and ideas still exist from older times in today versions. And personally, I think Oracle strategy is mainly on introducing new features (cloud drives development) and not robustness and fixing bugs. It’s also quite hard for developers to keep up with what hardware and software today are offering. Modern programming languages and remote direct memory access (RDMA) and persistent memory are just some examples. One goal is that hardware interacts the most direct way with hardware using the least amount of software (and main CPU cycles), so overall performance will improve. Dealing with Oracle RDBMS, licence costs that are CPU based may be lowered. Maybe one time you pay for payload and not CPU power?

On future

But maybe at the end the most fundamental steps are to

• release software using an appropriate open source licence
• work constantly on simplicity and close with end customers
• identify and lowering technical depths before implementing new features
• monetise not by selling licences but services

Back to reality

I think most developers, C-Managers and end users don’t care about databases at all. If it comes to Oracle RDBMS, few people understand the complex engine. Most workloads fit in standard configuration, but there are huge risk for misconfiguration. If in doubt, leave standard values and avoid to use hidden parameters. PL/SQL is far too complex, as a result most payload is simply being processed on application server or client level. To transport data has negative effects on costs, energy consumption and performance. On the other hand processing data outside database level may avoid database vendor lock-in. Application come and go, database engines stay.

An optimal service

• may be available promptly,
• usage may be self explaining (focus on usability),
• payload may stay under end user control (with the basic feature to erase) and
• it’s performance may scale automatically up and down according to current workload while maintaining low end users response times.

Easy requirements, hard to materialise.

Thinking out loud. What is your opinion?

Revision history:

2021-10-26

• Typos
• Intel® Xeon® 8352Y is 32 cores, not 16
  It’s a typo in Oracle X9M-x Whitepapers and documentation, I informed some people working for Oracle.
  Thank you, nirwander
• ExaC@C Experiment: Services will continue if you cap internet access, according to Thomas Teske.
  Thank you, Thomas
• Cloud learnings: ExaC@C mentioned as possible reason to minimise network latency
• Total cores comparison: Added storage cell XT

2021-10-28

• Hint why not persistent memory used on compute nodes.
• Mentioned other factors on CPUs to influence overall system performance
• X9 storage nodes: Xeon 8352Y uses 16 instead 32 cores as an Intel Speed Select Technology – Performance Profile is in place. May not be used to lower RDBMS license cost, mentioned other possibilities.

Thanks for explanation, Gavin Parish and Thomas Teske

L’article Exadata X9M release est apparu en premier sur dbi Blog.

Regarding Oracle Database Appliance, use the script odaeraser.py to securely erase. It’s a good practise to do this before re-imaging.

When to secure erase storage devices

It’s convenient to erase data that falls in security category “low” and “medium” as described by NIST (see this document, Figure 4-1 “Sanitization and Disposition Decision Flow”). However, if the data can be categorized “high”, it’s recommended to physically destroy all storage devices by shredding and buying new ones. German “Bundesamt für Sicherheit in der Informationstechnik”, known as BSI, recommends similar to NIST:

• First determine the level of data protection according to DIN 66399
• Then choose an appropriate erase method

Furthermore, BSI recommends using higher efforts on secure erase when storage devices are not used in the same scope or sold.
Personal recommendation: do not sell used storage devices (hard drives, ssd, memory, CPUs). Main reasons are

• You do not know what happens to sold hardware (even if securely erased). The combination of physical possession of hardware and time higher the risks that data believed to be erased securely is recovered and abused.
• Data is a valuable asset. It’s value is even higher if data is stored in databases: it’s potentially curated, therefore very valuable and even in perfectly structured form to work with. That’s why using a database to deal with large amount of data.

About data erasure and sanitization

On Wikipedia, one can find an accurate explanation:

Data erasure (sometimes referred to as data clearing, data wiping, or data destruction) is a software-based
method of overwriting the data that aims to completely destroy all electronic data residing on a hard disk
drive or other digital media by using zeros and ones to overwrite data onto all sectors of the device.
By overwriting the data on the storage device, the data is rendered unrecoverable
and achieves data sanitization.

According to the “Guidelines for Media Sanitization” published by NIST,

Sanitization is a process to render access totarget data (the data subject to the sanitization technique)
on the media infeasible for a given level of recovery effort.
...
Clear, Purge, and Destroy are actions that can be taken to sanitize media. The categories of sanitization
are defined as follows:
- Clear applies logical techniques to sanitize data in all user-addressable storage locations for
protection against simple non-invasive data recovery techniques; typically applied through the standard
Read and Write commands to the storage device, such as by rewriting with a new value or using a menu option
to reset the device to the factory state (where rewriting is not supported).
- Purge applies physical or logical techniques that render Target Data recovery infeasible using state of
the art laboratory techniques.
- Destroy renders Target Data recovery infeasible using state of the art laboratory techniques and results
in the subsequent inability to use the media for storage of data.


Oracle Database Appliance: Script odaeraser.py

Oracle Corporation provides the script odaeraser.py that will both clean and purge all data stored on mass storage devices on Oracle Database Appliance. The script is exhausting possibilities both on given software and hardware layer. It follows the minimum sanitization recommendations documented in the NIST guidelines for all devices that permanently storage data used in the Oracle Database Appliance. For details, see pages 32 to 38 of this document.

odaeraser.py is written in python programming language and works on ODA platforms from V2, t0 X8 (and will probably also work on upcoming hardware), both high availability and lite versions (S and M). An ODA consists of multiple types of mass storage devices such as hard disks, SSDs and NVMe storage devices. The script first detects the device types and uses then an appropriate cleaning and purging method.

Script usage can be found in Oracle documentation, for example for current X8 platform.

Script analysis

Done on script version 19.3.0.0.0/1 dated  2019/04/15
Issued erase commands and parameters did not change comparing to version 18.3.0.0.0/3 dated 2018/10/15.
Personally, I do not expect functional changes in script for future 19.x script updates.

Cleaning data

The script only cleans data on hard disk drives.

Method used for hard disk drives

See line 244 in script. Data on hard disk drives will be overwritten using scrub. In script, variables are used for parameters, what is finally executed is:

/usr/bin/scrub -p dod -f -b 10M $device

The parameters used are explained on the man page for scrub :

-p dod
Forces scrub to use a pattern to overwrite the disk drive following the standard
DoD 5220.22-M by the National Industrial Security Program, that is the nominal authority (in the United States) for managing the needs of private industry to access classified information.
The dod pattern overwrites drives in three steps using only:

1. 1. value 0, verify data written
   2. value 1, verify data written
   3. overwrite disk completely with random values, verify data written

-f
Scrub even if target contains signature indicating it has already been scrubbed.

-b <scrub block size>
Perform read and write calls using the specified block size (in bytes). K, M, or G may be appended to the number to change the units to KiBytes, MiBytes, or GiBytes, respectively. In script, 10MiBytes are used (line 84)

Purging Data

Methods used for hard disk drives and SSDs

The following methods are using SCSI commandos which are sent from the initiator (the host bus adapter) to the target (the disk or SSD). According to SBC-4 documentation on SCSI commandos (see this document, chapter 4.11 “sanitize operations”) , the sanitize commando on the storage device is resumed after a power loss or logical unit reset. During sanitization, the storage device does not accept new commandos from the initiator. Sanitization operations are performed on

• The medium on the device that store logical block data
• The medium on the device that is not being used to store logical block data. That are areas previously used to store block data, areas available for allocation and physical blocks that have become inaccessible.
• All cache

Crypto erase

See line 257 in script. On certain storage devices, a key is used to encrypt data. The crypto erase operation is removing this key and replacing it with a new key. Without the original key, it’s not possible to decrypt data on the device anymore. In script, variables are used for parameters, what is finally executed is:

/usr/bin/sg_raw -vvv -t 300 48 83 00 00 00 00 00 00 00 00
The sg_raw utility sends an arbitrary SCSI command (between 6 and 256 bytes) to the device. The command byte is specified in plain hex format without a prefix or suffix. The parameters used are explained on the man page for sg_raw :

-vvv
Increase level of verbosity

-t <timeout>
Wait up to # of seconds for command completion (in script: 300 seconds, see line 80). Note that if a command times out the operating system may start by aborting the command and if that is unsuccessful it may attempt to reset the device.

<scsi command>
SCSI command to sanitize device using crypto erase method (in script: 48 83 00 00 00 00 00 00 00 00, line 249)

Overwrite

See line 271  in script. To overwrite data on storage device, a pattern is used. In script, variables are used for parameters, what is finally executed is:

/usr/bin/sg_raw -vvv --infile /tmp/odaeraser.zeros.dat --send=516 48 81 00 00 00 00 00 02 04 00

-vvv
Increase level of verbosity

--infile /tmp/odaeraser.zeros.dat
Read data from File instead of stdin, here from file /tmp/odaeraser.zeros.dat

--send=516
Read SLEN bytes of data, either from stdin or from a file, and send them to the DEVICE

48 81 00 00 00 00 00 02 04 00
SCSI command to sanitize device using overwrite method

Block erase

See line 285 in script. Block erase causes the device to alter information by setting the physical blocks, even those not accessible by the operating system, to a vendor specific value. In script, variables are used for parameters, what is finally executed is:

/usr/bin/sg_raw -vvv 48 82 00 00 00 00 00 00 00 00

-vvv
Increase level of verbosity

48 82 00 00 00 00 00 00 00 00
SCSI command to sanitize device using block erase method

Method used to purge NVM Express SSDs

Secure erase

See line 296 in script. If the NVMe SSD is capable of self-encryption, the “crypto erase” method is used, otherwise it uses “block erase” method. In script, variables are used for parameters, what is finally executed is:

/usr/sbin/nvmeadm erase -a -s
The parameters used are explained on the man page of nvmeadm

erase
Erases the NVMe namespace media for the controller

-a
Erase all namespaces

-s
Retrieves the vendor-specific information of the specified controller

L’article When to secure erase storage devices? How to do it on Oracle Database Appliances? est apparu en premier sur dbi Blog.

Starting from version 19.11 of the Oracle Database Appliance software bundle, OLVM will definitely replace OVM server as virtualization engine. This change has quite a big impact as there will be no more virtualized specific deployment. Bare metal will come as a standard for all ODAs, and creating VMs for applications, or even databases, will be possible on top of that, for sure if you need them. And you can decide later to use virtualized machines. For now, 19.11 not being yet available, you’ll have to decide to deploy 19.10, not available for full virtualization support, and 19.9, the last release for OVM based virtualization. Which one should you use?

Past

OVM server stands for Oracle Virtualized Machines server. This comes on top of Xen, an 18-year old hypervisor also available in other commercial products, like Citrix XenServer. The first goal of OVM server on ODA was to allow all configurations of cores regarding the license (for example, bare metal X3-2 ODAs were only supporting 4+ cores, when virtualized mode started from 2 cores). The second goal was to benefit from the remaining cores on the system to run your applications, most often those connected to the databases. It made sense because ODAs were expensive servers, so buying a 2-node appliance with its dedicated storage for only using a pair of cores for a few databases was not that interesting.

In 2016, Oracle put “lite” ODAs on the market with a brand new CLI (odacli) and without virtualized mode. Virtualization was only possible on these servers with KVM, which is a virtualization tool provided within the Linux OS, but without the possibility to dedicate some cores to the database and the others to VMs. HA ODAs were still compatible with the OVM based virtualized mode, but only bare metal mode was using the brand new CLI.

Present

Oracle did a lot of work on ODA, especially regarding the odacli features. These features have not been back ported to oakcli, and oakcli is basically stuck to what’s were available 5 years ago. No GUI, no Data Guard feature, no backup management, no repository, no jobs, … Going back to oakcli after using odacli is a big step backward.

If you want to use virtualized machines on ODA, you have 2 options. Use the virtualized mode and get stuck with oakcli (and it’s also limited to HA ODAs) or use the bare metal mode, but hard partitioning is not supported for the VMs, meaning that you will run your applications on the cores dedicated to your DBs.

Virtualized mode is still available up to 19.9 version of the ODA software bundle. This software bundle not being too old, you could think that it’s the solution to adopt. Current bare metal version is 19.10.

Lite models brought huge popularity to ODAs, most of the customers are using these models. Some customers still need HA ODA, but most of them are deployed in bare metal mode. As a result, I think that more than 90% of running ODAs are using odacli stack.

Future

Days of OVM server are numbered. Starting a new project in version 19.9 on ODA X8-HA this year may not be a good idea. If this solution will do the job, you will probably need to patch your ODA for security reason during the next years, and this would probably be a problem. Because patches are only bundle patches, you patch everything or nothing.

OLVM (Oracle Linux Virtualization Manager), will completely replace OVM server starting from 19.11. OLVM adds hard partitioning (and much more) on top of KVM, meaning that it will have the same license flexibility as OVM server.

I heard about a migration path currently being prepared by Oracle for going from OVM server to OLVM, but it’s surely quite a tough job and I would definitely prefer to do a reimaging instead of a complex patching if I had to choose. And I don’t think that a virtualized ODA could get future patches: you’ll probably stay in 19.9 for a while.

Nobody will prevent you to start with OVM server and you could probably switch to OLVM later, but you’ll have to change quite a lot of procedures, as odacli and OLVM are totally different tools compared to oakcli and OVM server. This will imply a lot of work from your side.

In terms of support, OVM server premier support is already over, but in the particular case of ODA, it’s probably still supported for a year or two, as it’s part of your ODA configuration. But ask this question to Oracle to make sure.

Conclusion

For a new project, I would not go on virtualized ODA 19.9, I would better deploy my ODA in 19.10 bare metal mode, play with it and get trained on it in the next months, and upgrade to 19.11 as soon as it will be available. This is because I want to deploy technologies that will fit my needs for the next 5 years. OLVM with odacli and 19c is probably the most convincing bundle for that.

L’article Does it still make sense to deploy a virtualized HA ODA in 2021? est apparu en premier sur dbi Blog.

Have you ever cloned a database using RMAN?
Have you ever gotten the advice not to clone during daytime to avoid performance downgrade on the entire storage or network?
For a proof of concept, we wanted to evaluate a Pure Storage array in terms of feasibility and speed of database cloning by using storage-based snapshot technology.
First, I want to mention that PureStorage produces Storage devices in several sizes, all based on NVMe devices together with a high bandwidth connection and data compression.
All in all a “fast as hell” storage but this is not enough. There is additionally a snapshot technology which enables a snapshot-copy of a volume in no time, regardless of its size.
We wanted to see if it works as well for cloning Oracle databases.

Starting point

We set up:
• Two virtual machines on two ESX hosts, running RHEL 7.9
• Two sets of disk-groups to store the database files.
• A source database, running on machine 1, Oracle Release 19.10.
• Prepare machine 2 for a target database to be created by cloning

We had to install only very few extras on the virtual machines
• Install rpms, required by PureStorage
• Configure multipathing
• Update UDEV rules

About the Storage

The GUI allows comfortably to create volumes and assign them to groups and “Protection Groups”.

Connect the volumes with the hosts.

In vSphere Client, attach the volumes to the VMs

Configure the volumes on Linux OS

configure multipath

vi /etc/multipath.conf

multipaths {
 multipath { wwid 3624a9370F9C23FA49CFC431C00013CD5  alias  oradbi1-data }
 multipath { wwid 3624a9370F9C23FA49CFC431C00013CD7  alias  oradbi1-fra  }
 multipath { wwid 3624a9370F9C23FA49CFC431C00013CD6  alias oradbi1-redo  }
}

create filesystems on the volumes

mkfs.ext4 /dev/mapper/oradbi2-data
mkfs.ext4 /dev/mapper/3624a9370f9c23fa49cfc431c00013cda
mkfs.ext4 /dev/mapper/3624a9370f9c23fa49cfc431c00013cd9
mkfs.ext4 /dev/mapper/3624a9370f9c23fa49cfc431c00013cd8

enter the volumes in fstab
vi /etc/fstab

/dev/mapper/3624a9370f9c23fa49cfc431c00013cd5 /u02/oradata             ext4  defaults  0 0
/dev/mapper/3624a9370f9c23fa49cfc431c00013cd6 /u03/oradata             ext4  defaults  0 0
/dev/mapper/3624a9370f9c23fa49cfc431c00013cd7 /u90/fast_recovery_area  ext4  defaults  0 0

And finally mount them

mount -a

Now you can clone the database.

On the target server, shut down the database and unmount the volumes:

Create a snapshot

You can either use the GUI or CLI.
in the web-console, navigate to Storage -> Protection Groups
select the Protection Group you want to “snapshot”
select “create” from the overflow-menu
provide a “speaking” name for your first snapshot.

Next you copy the volume snapshots to their target counterparts:

That’s it !

On the CLI it looks even simpler:

Only two commands required: “purepgroup snap” and “purevol copy”
That’s all !

You can mount the copied volumes now and start the database on the target side:

You have probably noted that the copy is nothing else than a copy.

Means that your target-DB has still the same name as the source-DB.

Don’t worry! Changing the name and the DBID of a database, will be continued next week.
see my blog “Rename your DB clone completely“

L’article Clone a Terabyte Database in Seconds est apparu en premier sur dbi Blog.