To automatically scale down and up the OCPU on ExaCC cluster according to the CPU utilisation, we can use dynamic scaling. Dynamic scaling engine can be installed on each cluster nodes VM, from a remote machine as master or from container like doker or podman. We can also installed dynamic engine as a Grid HA resource. In this blog I will describe the installation, configuration and use of Dynamic Scaling on the VM from a single node cluster, as daemon configured as oracle linux service. In another blog I will describe how to install it as Grid HA resource.
Read more: How to automatically manage OCPU on ExaCC using Dynamic ScalingPros/cons of each installation
| Installation | Pros | Cons |
| Remote machine as master | No Dynamic Scaling engine is installed on the Cluster nodes VM | Need to use Dynamic Scaling remote plug-in One Dynamic Scaling engine can only manage one cluster |
| Docker or podman container | No Dynamic Scaling engine is installed on the Cluster nodes VM Several cluster can be managed from the same remote machine | Need to use Dynamic Scaling remote plug-in Need to know docker and podman technology Installation more complicated |
| Each VM Cluster | Easy installation and use of Dynamic scaling Very limit host impact | Dynamic Scaling engine installed on each Cluster nodes VM 2 engine might decide for the scaling up/down, but with RAC database should not be a problem |
| Grid HA Resource | Easy installation and use of Dynamic scaling Very limit host impact Only one node decide about the scaling up/down | Dynamic Scaling engine installed on each Cluster nodes VM |
At our customer we have decided to install Dynamic Scaling engine on each Cluster nodes VM but running through one Grid HA resource that can be easily relocated from one node to the others.
Internet access is mandatory from the Cluster nodes to get access to OCI resource.
Installation of Dynamic Scaling
Our cluster nodes are running Oracle Linux 8.8.
[root@ExaCC-cl08n1 ~]# cat /etc/oracle-release Oracle Linux Server release 8.8
The appropriate Dynamic Scaling package needs to be dowloaded from Doc ID 2719916.1, in our case, p36585874_202_Linux-x86-64.zip file.
We need to unzip it and to install the package.
[root@ExaCC-cl08n1 opc]# rpm -qa | grep -i dynam [root@ExaCC-cl08n1 ~]# cd /home/opc/mwagner/dynscal-rpm/ [root@ExaCC-cl08n1 dynscal-rpm]# ls p36585874_202_Linux-x86-64.zip [root@ExaCC-cl08n1 dynscal-rpm]# unzip /home/opc/mwagner/dynscal-rpm/p36585874_202_Linux-x86-64.zip Archive: /home/opc/mwagner/dynscal-rpm/p36585874_202_Linux-x86-64.zip inflating: readme_dynamicscaling-2.0.2-3.el8.x86_64.txt inflating: dynamicscaling-2.0.2-3.el8.x86_64.rpm [root@ExaCC-cl08n1 dynscal-rpm]# ls -ltrh total 28M -rw-rw-r-- 1 root root 14M May 6 13:57 dynamicscaling-2.0.2-3.el8.x86_64.rpm -rw-rw-r-- 1 root root 740 May 6 14:27 readme_dynamicscaling-2.0.2-3.el8.x86_64.txt -rw-r--r-- 1 opc opc 14M Jul 26 08:38 p36585874_202_Linux-x86-64.zip [root@ExaCC-cl08n1 dynscal-rpm]# rpm -i dynamicscaling-2.0.2-3.el8.x86_64.rpm warning: dynamicscaling-2.0.2-3.el8.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID c9c430a5: NOKEY dynamicscaling-2.0.2.3 binary has been installed on /opt/dynamicscaling succesfully! [root@ExaCC-cl08n1 dynscal-rpm]# rpm -qa | grep -i dynam dynamicscaling-2.0.2-3.x86_64
Install oci-cli
I have been installing oci-cli by downloading the offline installation package (oci-cli-3.43.2-Oracle-Linux-8-Offline.zip) from github project, maintained by Oracle Corp, https://github.com/oracle/oci-cli?tab=readme-ov-file.
[opc@ExaCC-cl08n1 ~]$ cd /home/opc/mwagner/dynscal-rpm/
[opc@ExaCC-cl08n1 dynscal-rpm]$ unzip -q oci-cli-3.43.2-Oracle-Linux-8-Offline.zip
[opc@ExaCC-cl08n1 dynscal-rpm]$ ls -ltrh
total 119M
-rw-rw-r-- 1 root root 14M May 6 13:57 dynamicscaling-2.0.2-3.el8.x86_64.rpm
-rw-rw-r-- 1 root root 740 May 6 14:27 readme_dynamicscaling-2.0.2-3.el8.x86_64.txt
drwxrwxr-x 3 opc opc 58 Jun 18 23:26 oci-cli-installation
-rw-r--r-- 1 opc opc 14M Jul 26 08:38 p36585874_202_Linux-x86-64.zip
-rw-r--r-- 1 opc opc 92M Jul 26 08:47 oci-cli-3.43.2-Oracle-Linux-8-Offline.zip
[opc@ExaCC-cl08n1 dynscal-rpm]$ cd oci-cli-installation
[opc@ExaCC-cl08n1 oci-cli-installation]$ python --version
Python 3.6.8
[opc@ExaCC-cl08n1 oci-cli-installation]$ bash install.sh --offline-install
******************************************************************************
You have started the OCI CLI Installer in interactive mode. If you do not wish
to run this in interactive mode, please include the --accept-all-defaults option.
If you have the script locally and would like to know more about
input options for this script, then you can run:
./install.sh -h
If you would like to know more about input options for this script, refer to:
https://github.com/oracle/oci-cli/blob/master/scripts/install/README.rst
******************************************************************************
Starting OCI CLI Offline Installation
Running install script.
python3 ./install.py --offline-install
-- Verifying Python version.
-- Python version 3.6.8 okay.
===> In what directory would you like to place the install? (leave blank to use '/home/opc/lib/oracle-cli'):
-- Creating directory '/home/opc/lib/oracle-cli'.
-- We will install at '/home/opc/lib/oracle-cli'.
===> In what directory would you like to place the 'oci' executable? (leave blank to use '/home/opc/bin'):
-- Creating directory '/home/opc/bin'.
-- The executable will be in '/home/opc/bin'.
===> In what directory would you like to place the OCI scripts? (leave blank to use '/home/opc/bin/oci-cli-scripts'):
-- Creating directory '/home/opc/bin/oci-cli-scripts'.
-- The scripts will be in '/home/opc/bin/oci-cli-scripts'.
-- Trying to use python3 venv.
-- Executing: ['/usr/bin/python3', '-m', 'venv', '/home/opc/lib/oracle-cli']
-- Executing: ['/home/opc/lib/oracle-cli/bin/pip', 'install', 'pip', '--upgrade', '--find-links', 'cli-deps/python36.html', '--no-index']
Collecting pip
Installing collected packages: pip
Found existing installation: pip 9.0.3
Uninstalling pip-9.0.3:
Successfully uninstalled pip-9.0.3
Successfully installed pip-21.3.1
-- Executing: ['/home/opc/lib/oracle-cli/bin/pip', 'install', 'oci_cli', '--find-links', 'cli-deps/python36.html', '--no-index', '--ignore-requires-python']
Looking in links: cli-deps/python36.html
Processing ./cli-deps/oci_cli-3.43.2-py3-none-any.whl
Processing ./cli-deps/certifi-2024.6.2-py3-none-any.whl
Processing ./cli-deps/terminaltables-3.1.10-py2.py3-none-any.whl
Processing ./cli-deps/pyOpenSSL-23.2.0-py3-none-any.whl
Processing ./cli-deps/PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Processing ./cli-deps/oci-2.128.2-py3-none-any.whl
Processing ./cli-deps/python_dateutil-2.9.0.post0-py2.py3-none-any.whl
Processing ./cli-deps/prompt_toolkit-3.0.29-py3-none-any.whl
Processing ./cli-deps/arrow-1.2.3-py3-none-any.whl
Processing ./cli-deps/pytz-2024.1-py2.py3-none-any.whl
Processing ./cli-deps/six-1.16.0-py2.py3-none-any.whl
Processing ./cli-deps/click-8.0.4-py3-none-any.whl
Processing ./cli-deps/jmespath-0.10.0-py2.py3-none-any.whl
Processing ./cli-deps/cryptography-40.0.2-cp36-abi3-manylinux_2_28_x86_64.whl
Processing ./cli-deps/importlib_metadata-4.8.3-py3-none-any.whl
Processing ./cli-deps/circuitbreaker-1.4.0.tar.gz
Preparing metadata (setup.py) ... done
Processing ./cli-deps/wcwidth-0.2.13-py2.py3-none-any.whl
Processing ./cli-deps/typing_extensions-4.1.1-py3-none-any.whl
Processing ./cli-deps/cffi-1.15.1-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Processing ./cli-deps/pycparser-2.21-py2.py3-none-any.whl
Processing ./cli-deps/zipp-3.6.0-py3-none-any.whl
Using legacy 'setup.py install' for circuitbreaker, since package 'wheel' is not installed.
Installing collected packages: pycparser, cffi, zipp, typing-extensions, six, cryptography, wcwidth, pytz, python-dateutil, pyOpenSSL, importlib-metadata, circuitbreaker, certifi, terminaltables, PyYAML, prompt-toolkit, oci, jmespath, click, arrow, oci-cli
Running setup.py install for circuitbreaker ... done
Successfully installed PyYAML-6.0.1 arrow-1.2.3 certifi-2024.6.2 cffi-1.15.1 circuitbreaker-1.4.0 click-8.0.4 cryptography-40.0.2 importlib-metadata-4.8.3 jmespath-0.10.0 oci-2.128.2 oci-cli-3.43.2 prompt-toolkit-3.0.29 pyOpenSSL-23.2.0 pycparser-2.21 python-dateutil-2.9.0.post0 pytz-2024.1 six-1.16.0 terminaltables-3.1.10 typing-extensions-4.1.1 wcwidth-0.2.13 zipp-3.6.0
===> Modify profile to update your $PATH and enable shell/tab completion now? (Y/n):
===> Enter a path to an rc file to update (file will be created if it does not exist) (leave blank to use '/home/opc/.bashrc'):
-- Backed up '/home/opc/.bashrc' to '/home/opc/.bashrc.backup'
-- Tab completion set up complete.
-- If tab completion is not activated, verify that '/home/opc/.bashrc' is sourced by your shell.
-- ** Run `exec -l $SHELL` to restart your shell. **
-- Installation successful.
Announcement
============
1. Interactive mode now available in CLI
Have you tried the new interactive features in OCI CLI yet? You can get started by typing `oci -i`.
Learn more by watching our informative video on YouTube -> https://www.youtube.com/watch?v=lX29Xw1Te54&ab_channel=OracleLearning
Also see https://docs.oracle.com/iaas/Content/API/SDKDocs/cliusing_topic-Using_Interactive_Mode.htm
============
-- Run the CLI with /home/opc/bin/oci --help
Configure oci config file
Before configuring oci, I have been creating a group and user with the OCI console, and imported the key using the add api key menu. I have been using a key that is already existing. This information will then be used to create the config file.
Here is the oci config file for the current node.
[opc@ExaCC-cl08n1 ~]$ cd .oci [opc@ExaCC-cl08n1 .oci]$ ls -lrh total 12K -rw------- 1 opc opc 322 Jul 31 2023 config -rw------- 1 opc opc 450 Jul 31 2023 exacc_dbaas_automation_public.pem -rw------- 1 opc opc 1.7K Jul 31 2023 exacc_dbaas_automation.pem [opc@ExaCC-cl08n1 .oci]$ vi config [opc@ExaCC-cl08n1 .oci]$ cat config [DEFAULT] user=ocid1.user.oc1..aaaaaaaaz*****************************z2kndq fingerprint=fc:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:84 tenancy=ocid1.tenancy.oc1..aaaaaaaa52q*******************************wka region=eu-zurich-1 key_file=~/.oci/exacc_dbaas_automation.pem
Test oci cli
We can list all region in order to ensure our oci cli configuration is working.
[opc@ExaCC-cl08n1 ~]$ oci iam region list --output table /home/opc/lib/oracle-cli/lib64/python3.6/site-packages/oci/_vendor/httpsig_cffi/sign.py:10: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6. from cryptography.hazmat.backends import default_backend # noqa: F401 +-----+-------------------+ | key | name | +-----+-------------------+ | AMS | eu-amsterdam-1 | | ARN | eu-stockholm-1 | | AUH | me-abudhabi-1 | | BOG | sa-bogota-1 | | BOM | ap-mumbai-1 | | CDG | eu-paris-1 | | CWL | uk-cardiff-1 | | DXB | me-dubai-1 | | FRA | eu-frankfurt-1 | | GRU | sa-saopaulo-1 | | HYD | ap-hyderabad-1 | | IAD | us-ashburn-1 | | ICN | ap-seoul-1 | | JED | me-jeddah-1 | | JNB | af-johannesburg-1 | | KIX | ap-osaka-1 | | LHR | uk-london-1 | | LIN | eu-milan-1 | | MAD | eu-madrid-1 | | MEL | ap-melbourne-1 | | MRS | eu-marseille-1 | | MTY | mx-monterrey-1 | | MTZ | il-jerusalem-1 | | NRT | ap-tokyo-1 | | ORD | us-chicago-1 | | PHX | us-phoenix-1 | | QRO | mx-queretaro-1 | | SCL | sa-santiago-1 | | SIN | ap-singapore-1 | | SJC | us-sanjose-1 | | SYD | ap-sydney-1 | | VAP | sa-valparaiso-1 | | VCP | sa-vinhedo-1 | | XSP | ap-singapore-2 | | YNY | ap-chuncheon-1 | | YUL | ca-montreal-1 | | YYZ | ca-toronto-1 | | ZRH | eu-zurich-1 | +-----+-------------------+
Policy
In order to use Dynamic Scaling I have been added following OCI policy to the group.
Allow group exacc_dynamic_scaling to use exadata-infrastructures IN COMPARTMENT ExaCC_DBaaS
Allow group exacc_dynamic_scaling to use vmclusters IN COMPARTMENT ExaCC_DBaaS
Test Dynamic Scaling
We can now test Dynamic Scaling with the check option and getocpu option (to get current ocpu).
Test with check option is successful:
[opc@ExaCC-cl08n1 .oci]$ /opt/dynamicscaling/dynamicscaling.bin check \ > --ocicli \ > --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5h*****************************************j3ba \ > --ociprofile DEFAULT │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ Dynamicscaling for ExaCC & ExaCS - Version: 2.0.2-03 Copyright (c) 2020-2024 Oracle and/or its affiliates. ---------------------------------------------------------- Author: Ruggero Citton RAC Pack, Cloud Innovation and Solution Engineering Team │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ INFO: 2024-07-09 14:28:31: Checking OCI connectivity... SUCCESS: 2024-07-09 14:28:32: OCI connectivity check done successfully
getocpu option is also working and giving us current OCPU to be equal to 2 OCPU, 1 VM, so 2 per VM.
[opc@ExaCC-cl08n1 .oci]$ /opt/dynamicscaling/dynamicscaling.bin getocpu \ > --ocicli \ > --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5he*****************************************j3ba \ > --ociprofile DEFAULT │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ Dynamicscaling for ExaCC & ExaCS - Version: 2.0.2-03 Copyright (c) 2020-2024 Oracle and/or its affiliates. ---------------------------------------------------------- Author: Ruggero Citton RAC Pack, Cloud Innovation and Solution Engineering Team │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ INFO: 2024-07-09 14:29:54: Checking DB System status - ------------------------------------------------------- VM Cluster OCID : ocid1.vmcluster.oc1.eu-zurich-1.an5h*****************************************j3ba OCI profile name : DEFAULT - ------------------------------------------------------- DB System status : AVAILABLE Current OCPU : 2 Current physical CPUs : 2 - ------------------------------------------------------- Dynamicscaling Log file: '/tmp/dynamicscaling.log' ---------------------------------------------------------
Functionning
From the DocId, we can read that “Oracle DynamicScaling can be executed as standalone executable or as daemon on one or more ExaDB-D compute nodes or ExaDB-C@C vmcluster nodes. By default DynamicScaling is monitoring the CPUs with very limited host impact and if the load goes over the Maximum CPU threshold (“–maxthreshold”) for an interval of time (“–interval”), it will automatically will scale-up the OCPU by a factor (“–ocpu”) till a maximum limit (“–maxocpu”). If the load goes under the Minimum CPU threshold (“–minthreshold”) for an interval of time (“–interval”) scale down will be executed util the minimum limit (“–minocpu”) of ocpu.”
To avoid too frequent scale up and down, Dynamic Scaling will wait for one hour after having scaled up the OCPU before scaling it down again.
Minimun 4 OCPU factor for half rack.
Mininum 2 OCPU factor for quater rack.
Log file rotation
I have configured log file rotation.
First I have created needed log directory.
[opc@ExaCC-cl08n1 ~]$ sudo mkdir -p /acfs01/dynscal_logs/node1 [opc@ExaCC-cl08n1 ~]$ sudo chown -R opc: /acfs01/dynscal_logs
Then I have configured log file rotation as following:
[root@ExaCC-cl08n1 ~]# cd /etc/logrotate.d/
[root@ExaCC-cl08n1 logrotate.d]# ls -ltrh
total 76K
-rw-r--r-- 1 root root 145 Feb 19 2018 wtmp
-rw-r--r-- 1 root root 408 Mar 7 2019 psacct
-rw-r--r-- 1 root root 172 Mar 11 2021 iscsiuiolog
-rw-r--r-- 1 root root 88 Apr 12 2021 dnf
-rw-r--r-- 1 root root 160 Dec 16 2021 chrony
-rw-r--r-- 1 root root 155 Feb 7 2022 aide
-rw-r--r-- 1 root root 91 Apr 11 2022 bootlog
-rw-r--r-- 1 root root 67 Dec 2 2022 ipmctl
-rw-r--r-- 1 root root 130 Mar 31 2023 btmp
-rw-r--r-- 1 root root 93 Sep 21 2023 firewalld
-rw-r--r-- 1 root root 226 Oct 25 2023 syslog
-rw-r--r-- 1 root root 237 Feb 22 15:50 sssd
-rw-r--r-- 1 root root 155 Mar 7 05:29 samba
-rw-r--r-- 1 root root 125 Apr 11 07:00 nscd
-rw-r--r-- 1 root root 103 May 14 11:51 oraiosaudit
-rw-r--r-- 1 root root 103 May 14 11:51 oraasmaudit
-rw-r--r-- 1 root root 103 May 14 11:51 oraapxaudit
-rw-r--r-- 1 root root 89 May 14 12:18 asmaudit
-rw-r--r-- 1 root root 172 May 14 14:06 ora_rdbms_uniaud
[root@ExaCC-cl08n1 logrotate.d]# vi dynamicscaling
[root@ExaCC-cl08n1 logrotate.d]# cat dynamicscaling
/acfs01/dynscal_logs/node1/* {
su opc opc
daily
notifempty
missingok
sharedscripts
copytruncate
rotate 10
size 30M
compress
}
Create linux service for Dynamic Scaling
I have created a linux service to have Dynamic Scaling running automatically as a daemon.
[root@ExaCC-cl08n1 ~]# vi /etc/systemd/system/dynamicscaling.service [root@ExaCC-cl08n1 ~]# cat /etc/systemd/system/dynamicscaling.service [Unit] Description=Dynamicscaling Wants=network-online.target local-fs.target After=network-online.target local-fs.target [Service] User=opc Type=simple Environment="HTTP_PROXY=http://webproxy.domain.com:XXXX" Environment="HTTPS_PROXY=http://webproxy.domain.com:XXXX" Environment="http_proxy=http://webproxy.domain.com:XXXX" Environment="https_proxy=http://webproxy.domain.com:XXXX" Environment="PATH=/sbin:/bin:/usr/sbin:/usr/bin:/home/opc/bin:/home/opc/.local/bin" ExecStart=/bin/sh -c "/opt/dynamicscaling/dynamicscaling.bin --ocicli --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5he***********************************j3ba --ociprofile DEFAULT --interval 60 --maxthreshold 70 --minthreshold 40 --maxocpu 12 --minocpu 4 --ocpu 4 --logpath /acfs01/dynscal_logs/node1" TimeoutStartSec=300 PIDFile=/tmp/.dynamicscaling.pid Restart=on-failure RestartSec=5s ExecStop=/bin/kill -s SIGINT $MAINPID [Install] WantedBy=multi-user.target [root@ExaCC-cl08n1 ~]# systemctl daemon-reload [root@ExaCC-cl08n1 ~]# systemctl enable dynamicscaling.service Created symlink /etc/systemd/system/multi-user.target.wants/dynamicscaling.service → /etc/systemd/system/dynamicscaling.service.
Linux service has been tested.
[root@ExaCC-cl08n1 ~]# ps -ef | grep -i [d]ynamic
[root@ExaCC-cl08n1 ~]# systemctl start dynamicscaling.service
[root@ExaCC-cl08n1 ~]# systemctl status dynamicscaling.service
● dynamicscaling.service - Dynamicscaling
Loaded: loaded (/etc/systemd/system/dynamicscaling.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2024-07-09 22:47:16 CEST; 2s ago
Main PID: 272574 (/opt/dynamicsca)
Tasks: 2 (limit: 319999)
Memory: 119.6M
CGroup: /system.slice/dynamicscaling.service
├─272574 /opt/dynamicscaling/dynamicscaling.bin --ocicli --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5h*****************************************j3ba >
└─272590 /home/opc/lib/oracle-cli/bin/python3 /home/opc/bin/oci db vm-cluster get --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5h*****************************************j3ba --profile DEFAULT
Jul 09 22:47:17 ExaCC-cl08n1 sh[272574]: │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
Jul 09 22:47:17 ExaCC-cl08n1 sh[272574]: Dynamicscaling for ExaCC & ExaCS - Version: 2.0.2-03
...
...
...
[root@ExaCC-cl08n1 ~]# ps -ef | grep -i [d]ynamic
opc 272677 1 0 22:47 ? 00:00:00 /opt/dynamicscaling/dynamicscaling.bin --ocicli --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5h*****************************************3ba --ociprofile DEFAULT --interval 60 --maxthreshold 70 --minthreshold 40 --maxocpu 24 --minocpu 4 --ocpu 4
[root@ExaCC-cl08n1 ~]# systemctl stop dynamicscaling.service
[root@ExaCC-cl08n1 ~]# systemctl status dynamicscaling.service
● dynamicscaling.service - Dynamicscaling
Loaded: loaded (/etc/systemd/system/dynamicscaling.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2024-07-09 22:47:32 CEST; 2s ago
Process: 273462 ExecStop=/bin/kill -s SIGINT $MAINPID (code=exited, status=0/SUCCESS)
Process: 272677 ExecStart=/bin/sh -c /opt/dynamicscaling/dynamicscaling.bin --ocicli --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5h*****************************************j3ba --ociprofile DEFAULT --i>
Main PID: 272677 (code=exited, status=1/FAILURE)
Jul 09 22:47:17 ExaCC-cl08n1 sh[272574]: Author: Ruggero Citton
Jul 09 22:47:17 ExaCC-cl08n1 sh[272574]: RAC Pack, Cloud Innovation and Solution Engineering Team
Jul 09 22:47:17 ExaCC-cl08n1 sh[272574]: │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
Jul 09 22:47:17 ExaCC-cl08n1 sh[272574]: INFO: 2024-07-09 22:47:17: ================================================
Jul 09 22:47:17 ExaCC-cl08n1 sh[272574]: INFO: 2024-07-09 22:47:17: Dynamicscaling version '2.0.2-03' is starting...
Jul 09 22:47:17 ExaCC-cl08n1 sh[272574]: INFO: 2024-07-09 22:47:17: ================================================
Jul 09 22:47:32 ExaCC-cl08n1 systemd[1]: Stopping Dynamicscaling...
Jul 09 22:47:32 ExaCC-cl08n1 systemd[1]: dynamicscaling.service: Main process exited, code=exited, status=1/FAILURE
Jul 09 22:47:32 ExaCC-cl08n1 systemd[1]: dynamicscaling.service: Failed with result 'exit-code'.
Jul 09 22:47:32 ExaCC-cl08n1 systemd[1]: Stopped Dynamicscaling.
[root@ExaCC-cl08n1 ~]# ps -ef | grep -i [d]ynamic
[root@ExaCC-cl08n1 ~]#
Test Dynamic Scaling scale down
The OCPU has been increased to 12 OCPU. We have only one nodes in this cluster so 12 OCP per VM.
Check current OCUP.
[opc@ExaCC-cl08n1 ~]$ /opt/dynamicscaling/dynamicscaling.bin getocpu \ > --ocicli \ > --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5h*****************************************j3ba \ > --ociprofile DEFAULT │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ Dynamicscaling for ExaCC & ExaCS - Version: 2.0.2-03 Copyright (c) 2020-2024 Oracle and/or its affiliates. ---------------------------------------------------------- Author: Ruggero Citton RAC Pack, Cloud Innovation and Solution Engineering Team │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ INFO: 2024-07-09 22:54:31: Checking DB System status - ------------------------------------------------------- VM Cluster OCID : ocid1.vmcluster.oc1.eu-zurich-1.an5h*****************************************j3ba OCI profile name : DEFAULT - ------------------------------------------------------- DB System status : AVAILABLE Current OCPU : 12 Current physical CPUs : 12 - ------------------------------------------------------- Dynamicscaling Log file: '/tmp/dynamicscaling.log' ---------------------------------------------------------
Let’s start Dynamic Scaling service.
[root@ExaCC-cl08n1 ~]# ps -ef | grep -i [d]ynamic [root@ExaCC-cl08n1 ~]# systemctl start dynamicscaling.service [root@ExaCC-cl08n1 ~]# ps -ef | grep -i [d]ynamic opc 348304 1 0 23:07 ? 00:00:00 /opt/dynamicscaling/dynamicscaling.bin --ocicli --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5he*****************************************j3ba --ociprofile DEFAULT --interval 60 --maxthreshold 70 --minthreshold 40 --maxocpu 24 --minocpu 4 --ocpu 4
Check the logs.
[opc@ExaCC-cl08n1 ~]$ tail -f /acfs01/dynscal_logs/node1dynamicscaling.log 2024-07-09 23:07:49: Getting lifecycle-state 2024-07-09 23:07:50: DB System status......: AVAILABLE 2024-07-09 23:07:50: Resetting consecutive DB System 'UPDATING' status count 2024-07-09 23:07:50: Checking current core count 2024-07-09 23:07:50: Getting cpu core count for 'ocid1.vmcluster.oc1.eu-zurich-1.an5h*****************************************j3ba' with oci-cli 2024-07-09 23:07:50: Running on ExaCC getting cpus enabled 2024-07-09 23:07:50: Current OCPU=12 2024-07-09 23:07:52: Local host load ......: 1.5 2024-07-09 23:07:52: CPU usage is under minthreshold 2024-07-09 23:07:52: Next measure in about 60 secs... 2024-07-09 23:08:52: Checking current core count 2024-07-09 23:08:52: Getting cpu core count for 'ocid1.vmcluster.oc1.eu-zurich-1.an5h*****************************************j3ba' with oci-cli 2024-07-09 23:08:53: Running on ExaCC getting cpus enabled 2024-07-09 23:08:53: Current OCPU=12 2024-07-09 23:08:55: Local host load ......: 0.9 2024-07-09 23:08:55: Current load is under/equal minimum threshold '40' for '60' secs 2024-07-09 23:08:55: Checking DB System status 2024-07-09 23:08:55: Getting lifecycle-state 2024-07-09 23:08:56: DB System status......: AVAILABLE 2024-07-09 23:08:56: Resetting consecutive DB System 'UPDATING' status count 2024-07-09 23:08:56: Requesting OCPU scale-Down by a factor of '4' 2024-07-09 23:08:56: Scaling-down the core-count... 2024-07-09 23:08:57: Scaling-down in progress, sleeping 180 secs... 2024-07-09 23:11:57: Resetting consecutive scale-up operation count 2024-07-09 23:11:57: zzzzzz
Monitoring Dynamic Scaling
Monitoring Dynamic Scaling, I could also see some scale up in the logs.
Threshold value used in that case:
2024-07-19 18:16:17: - Dynamicscaling is running as daemon with pid '395012'
2024-07-19 18:16:17: VM Cluster OCID : ocid1.vmcluster.oc1.eu-zurich-1.anXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXj5gq
2024-07-19 18:16:17: OCI-Client Profile : DEFAULT
2024-07-19 18:16:17: DB System shape : ExadataCC.X9M
2024-07-19 18:16:17: Maximum OCPU Number : 48
2024-07-19 18:16:17: Minimum OCPU Number : 4
2024-07-19 18:16:17: OCPU scale factor : 4
2024-07-19 18:16:17: Incremental scale-up : DISABLE
2024-07-19 18:16:17: Measure interval : 60
2024-07-19 18:16:17: Maximum Load threshold : 75
2024-07-19 18:16:17: Minimum Load threshold : 60
2024-07-19 18:16:17: Scale-down wait time : 60 minutes
2024-07-19 18:16:17: Cluster-aware : DISABLE
2024-07-19 18:16:17: Dry-run : DISABLE
No action:
2024-07-19 18:25:41: Local host load ……: 67.8
2024-07-19 18:25:41: Current load is between minimum '60' and maximun '75' threshold, no actions taken
Scale up:
2024-07-19 18:51:28: Local host load ……: 99.4
2024-07-19 18:51:28: Current load is over/equal maximun threshold '75' for '60' secs
2024-07-19 18:51:28: Checking DB System status
2024-07-19 18:51:28: Getting lifecycle-state
2024-07-19 18:51:31: DB System status……: AVAILABLE
2024-07-19 18:51:31: Resetting consecutive DB System 'UPDATING' status count
2024-07-19 18:51:31: Requesting OCPU scale-Up by a factor of '4'
2024-07-19 18:51:31: Scaling-up the core-count
Scaling down not possible for the next 60 min following a scale-up:
2024-07-19 18:56:52: Scaling-Down currently not possible, waiting '2024-07-19 19:51:31', 60 minutes after latest scale-Up time
Setocpu with Dynamic Scaling
With Dynamic Scaling we can also set the OCPU.
Get current OCPU :
[opc@ExaCC-cl08n1 ~]$ /opt/dynamicscaling/dynamicscaling.bin getocpu --ocicli --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5he*****************************************j3ba --ociprofile DEFAULT │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ Dynamicscaling for ExaCC & ExaCS - Version: 2.0.2-03 Copyright (c) 2020-2024 Oracle and/or its affiliates. ---------------------------------------------------------- Author: Ruggero Citton RAC Pack, Cloud Innovation and Solution Engineering Team │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ INFO: 2024-07-19 08:06:02: Checking DB System status - ------------------------------------------------------- VM Cluster OCID : ocid1.vmcluster.oc1.eu-zurich-1.an5he*****************************************j3ba OCI profile name : DEFAULT - ------------------------------------------------------- DB System status : UPDATING Current OCPU : 4 Current physical CPUs : 4 - ------------------------------------------------------- Dynamicscaling Log file: '/tmp/dynamicscaling.log' ---------------------------------------------------------
Set OCPU to 16:
[opc@ExaCC-cl08n1 ~]$ /opt/dynamicscaling/dynamicscaling.bin setocpu --ocicli --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5he*****************************************j3ba --ociprofile DEFAULT --ocpu 16 │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ Dynamicscaling for ExaCC & ExaCS - Version: 2.0.2-03 Copyright (c) 2020-2024 Oracle and/or its affiliates. ---------------------------------------------------------- Author: Ruggero Citton RAC Pack, Cloud Innovation and Solution Engineering Team │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ INFO: 2024-07-19 08:05:23: dynamicscaling log file at '/tmp/dynamicscaling.log' INFO: 2024-07-19 08:05:26: Checking DB System status INFO: 2024-07-19 08:05:28: Current OCPU=4 SUCCESS: 2024-07-19 08:05:29: Scaling-up to OCPU=16 in progress, please wait...
Check by getting ocpu with Dynamic Scaling:
[opc@ExaCC-cl08n1 ~]$ /opt/dynamicscaling/dynamicscaling.bin getocpu --ocicli --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5he*****************************************j3ba --ociprofile DEFAULT │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ Dynamicscaling for ExaCC & ExaCS - Version: 2.0.2-03 Copyright (c) 2020-2024 Oracle and/or its affiliates. ---------------------------------------------------------- Author: Ruggero Citton RAC Pack, Cloud Innovation and Solution Engineering Team │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ INFO: 2024-07-19 08:08:40: Checking DB System status - ------------------------------------------------------- VM Cluster OCID : ocid1.vmcluster.oc1.eu-zurich-1.an5he*****************************************j3ba OCI profile name : DEFAULT - ------------------------------------------------------- DB System status : UPDATING Current OCPU : 16 Current physical CPUs : 16 - ------------------------------------------------------- Dynamicscaling Log file: '/tmp/dynamicscaling.log' ---------------------------------------------------------
We have now 16 OCPU.
And in the log I could already see it was immediately scaled down by Dynamic Scaling daemon. This is why, when we have Dynamic Scaling running, there is no sense to manual increase OCPU. Dynamic Scaling daemon will do it for us.
[opc@ExaCC-cl08n1 ~]$ tail -f /acfs01/dynscal_logs/node1/dynamicscaling.log 2024-07-19 08:08:19: Current OCPU=16 2024-07-19 08:08:21: Local host load ......: 1.1 2024-07-19 08:08:21: Current load is under/equal minimum threshold '40' for '60' secs 2024-07-19 08:08:21: Checking DB System status 2024-07-19 08:08:21: Getting lifecycle-state 2024-07-19 08:08:22: DB System status......: AVAILABLE 2024-07-19 08:08:22: Resetting consecutive DB System 'UPDATING' status count 2024-07-19 08:08:22: Requesting OCPU scale-Down by a factor of '4' 2024-07-19 08:08:22: Scaling-down the core-count... 2024-07-19 08:08:23: Scaling-down in progress, sleeping 180 secs...
Scheduling
With Dynamic Scaling we can schedule weekly or long term period. The scheduling has got the priority over the load mesure.
Let’s schedule a weekly schedule for Friday between 9am and 4pm to have 16 OCPU. This will be done with the option --scheduling "Friday:9-16:16". Of course we can not change scheduling while the daemon is running as the scheduling is part of the daemon.
Get current OCPU:
[opc@ExaCC-cl08n1 ~]$ /opt/dynamicscaling/dynamicscaling.bin getocpu --ocicli --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5he*****************************************j3ba --ociprofile DEFAULT │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ Dynamicscaling for ExaCC & ExaCS - Version: 2.0.2-03 Copyright (c) 2020-2024 Oracle and/or its affiliates. ---------------------------------------------------------- Author: Ruggero Citton RAC Pack, Cloud Innovation and Solution Engineering Team │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ INFO: 2024-07-19 07:54:12: Checking DB System status - ------------------------------------------------------- VM Cluster OCID : ocid1.vmcluster.oc1.eu-zurich-1.an5he*****************************************j3ba OCI profile name : DEFAULT - ------------------------------------------------------- DB System status : AVAILABLE Current OCPU : 4 Current physical CPUs : 4 - ------------------------------------------------------- Dynamicscaling Log file: '/tmp/dynamicscaling.log' ---------------------------------------------------------
We have currently 4 OCPU.
Let’s stop the daemon, change its configuration to have the scheduling option as well and start it again.
[opc@ExaCC-cl08n1 ~]$ sudo systemctl stop dynamicscaling.service [opc@ExaCC-cl08n1 ~]$ ps -ef | grep -i [d]ynamic [opc@ExaCC-cl08n1 ~]$ [opc@ExaCC-cl08n1 ~]$ sudo vi /etc/systemd/system/dynamicscaling.service [opc@ExaCC-cl08n1 ~]$ sudo systemctl daemon-reload [opc@ExaCC-cl08n1 ~]$ cat /etc/systemd/system/dynamicscaling.service [Unit] Description=Dynamicscaling Wants=network-online.target local-fs.target After=network-online.target local-fs.target [Service] User=opc Type=simple Environment="HTTP_PROXY=http://webproxy.domain.com:XXXX" Environment="HTTPS_PROXY=http://webproxy.domain.com:XXXX" Environment="http_proxy=http://webproxy.domain.com:XXXX" Environment="https_proxy=http://webproxy.domain.com:XXXX" Environment="PATH=/sbin:/bin:/usr/sbin:/usr/bin:/home/opc/bin:/home/opc/.local/bin" ExecStart=/bin/sh -c "/opt/dynamicscaling/dynamicscaling.bin --ocicli --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5he*****************************************j3ba --ociprofile DEFAULT --interval 60 --maxthreshold 70 --minthreshold 40 --maxocpu 12 --minocpu 4 --ocpu 4 --logpath /acfs01/dynscal_logs/node1 --scheduling 'Friday:9-16:16'" TimeoutStartSec=300 PIDFile=/tmp/.dynamicscaling.pid Restart=on-failure RestartSec=5s ExecStop=/bin/kill -s SIGINT $MAINPID [Install] WantedBy=multi-user.target [opc@ExaCC-cl08n1 ~]$ [opc@ExaCC-cl08n1 ~]$ ps -ef | grep -i [d]ynamic [opc@ExaCC-cl08n1 ~]$ sudo systemctl start dynamicscaling.service [opc@ExaCC-cl08n1 ~]$ ps -ef | grep -i [d]ynamic opc 278950 1 26 08:30 ? 00:00:00 /opt/dynamicscaling/dynamicscaling.bin --ocicli --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5he*****************************************j3ba --ociprofile DEFAULT --interval 60 --maxthreshold 70 --minthreshold 40 --maxocpu 12 --minocpu 4 --ocpu 4 --logpath /acfs01/dynscal_logs/node1 --scheduling Friday:9-16:16 [opc@ExaCC-cl08n1 ~]$
It is not already 9am, so OCPU are still configured with 4 OCPU.
[opc@ExaCC-cl08n1 ~]$ date Fri Jul 19 08:31:36 CEST 2024 [opc@ExaCC-cl08n1 ~]$ /opt/dynamicscaling/dynamicscaling.bin getocpu --ocicli --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5he*****************************************j3ba --ociprofile DEFAULT │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ Dynamicscaling for ExaCC & ExaCS - Version: 2.0.2-03 Copyright (c) 2020-2024 Oracle and/or its affiliates. ---------------------------------------------------------- Author: Ruggero Citton RAC Pack, Cloud Innovation and Solution Engineering Team │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ INFO: 2024-07-19 08:32:36: Checking DB System status - ------------------------------------------------------- VM Cluster OCID : ocid1.vmcluster.oc1.eu-zurich-1.an5he*****************************************j3ba OCI profile name : DEFAULT - ------------------------------------------------------- DB System status : AVAILABLE Current OCPU : 4 Current physical CPUs : 4 - ------------------------------------------------------- Dynamicscaling Log file: '/tmp/dynamicscaling.log' ---------------------------------------------------------
In the logs, we can now see such message, if we are out of the scheduling time.
[opc@ExaCC-cl08n1 ~]$ grep -i weekly /acfs01/dynscal_logs/node1/dynamicscaling.log 2024-07-19 08:30:57: No applicable OCPU weekly scheduling found 2024-07-19 08:32:04: No applicable OCPU weekly scheduling found 2024-07-19 08:33:10: No applicable OCPU weekly scheduling found 2024-07-19 08:34:17: No applicable OCPU weekly scheduling found 2024-07-19 08:35:24: No applicable OCPU weekly scheduling found
Now we are in the scheduling, on Friday after 9am and before 5pm. And we can see that the OCPU have been increased.
[opc@ExaCC-cl08n1 ~]$ date Fri Jul 19 09:04:29 CEST 2024 [opc@ExaCC-cl08n1 ~]$ /opt/dynamicscaling/dynamicscaling.bin getocpu --ocicli --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5he*****************************************j3ba --ociprofile DEFAULT │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ Dynamicscaling for ExaCC & ExaCS - Version: 2.0.2-03 Copyright (c) 2020-2024 Oracle and/or its affiliates. ---------------------------------------------------------- Author: Ruggero Citton RAC Pack, Cloud Innovation and Solution Engineering Team │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│ INFO: 2024-07-19 09:04:32: Checking DB System status - ------------------------------------------------------- VM Cluster OCID : ocid1.vmcluster.oc1.eu-zurich-1.an5he*****************************************j3ba OCI profile name : DEFAULT - ------------------------------------------------------- DB System status : AVAILABLE Current OCPU : 16 Current physical CPUs : 16 - ------------------------------------------------------- Dynamicscaling Log file: '/tmp/dynamicscaling.log' ---------------------------------------------------------
And now we will find such message in the log telling us we are in the configured scheduled time.
[opc@ExaCC-cl08n1 ~]$ tail -f /acfs01/dynscal_logs/node1/dynamicscaling.log 2024-07-19 09:35:18: Current OCPU=16 2024-07-19 09:35:18: OCPU=16 is defined by scheduling 2024-07-19 09:35:20: Local host load ......: 0.5 2024-07-19 09:35:20: Current OCPU=16 is equal to scheduled OCPU, no action taken
To wrap up
Dynamic Scaling is a great tool that will help to automatically manage OCPU. We will not pay for non used OCPU and we will have enough OCPU if needed. Also we can configure specific scheduling for OCPU. Following DocId from Oracle MOS might be interesting to review for further details:
- (ODyS) Oracle Dynamic Scaling engine – Scale-up and Scale-down automation utility for OCI DB System (ExaDB-D/ExaDB-C@C) (Doc ID 2719916.1)
- (ODyS) Oracle Dynamic Scaling – Remote Plug-in (Doc ID 2770544.1)
- (ODyS) How to make Oracle Dynamic Scaling an HA cluster resource (Doc ID 2834931.1)
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/MAW_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/JDU_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/11/CBL_web-min-min-scaled.jpg)