Issue running Dynamic Scaling on hardened system

During my Dynamic Scaling tests, you might have seen my other blog on same subject, I faced one issue where Dynamic Scaling could not be started properly due to hardened configuration. In this blog I would like to share my investigation and solution on this subject.

Problem description

I have installed Dynamic Scaling package and remote plug-in on a Red Hat VM, in the intention to be used as remote master vm for Dynamic Scaling.

[root@dynscal-vm ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.9 (Ootpa)

[root@dynscal-vm dynamscaling-rpm]# rpm -i dynamicscaling-2.0.2-3.el8.x86_64.rpm
warning: dynamicscaling-2.0.2-3.el8.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID c9c430a5: NOKEY

dynamicscaling-2.0.2.3 binary has been installed on /opt/dynamicscaling succesfully!

[root@dynscal-vm dynamscaling-rpm]# rpm -i dynamicscaling-plugin-1.0.1-13.el8.x86_64.rpm
warning: dynamicscaling-plugin-1.0.1-13.el8.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID c9c430a5: NOKEY

dynamicscaling-plugin-1.0.1.13 binary has been installed on /opt/dynamicscaling-plugin succesfully!

I have been installing other required package.

[root@dynscal-vm yum.repos.d]# dnf install glibc-langpack-en

And then I tried to start manually Dynamic Scaling daemon.

[root@dynscal-vm ~]# cd /opt/dynamicscaling
[root@dynscal-vm dynamicscaling]# ./dynamicscaling.bin status
[root@dynscal-vm dynamicscaling]#

Which did not work. Same can be seen when displaying the status.

[root@dynscal-vm ~]# /opt/dynamicscaling/dynamicscaling.bin status

│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
 Dynamicscaling for ExaCC & ExaCS - Version: 2.0.2-03
 Copyright (c) 2020-2024 Oracle and/or its affiliates.
----------------------------------------------------------
 Author: Ruggero Citton 
 RAC Pack, Cloud Innovation and Solution Engineering Team
│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│

Dynamicscaling is not running

And same problem with the remote plug-in.

[root@dynscal-vm dynamicscaling-plugin]# /opt/dynamicscaling-plugin/dynamicscaling-plugin.bin --ocicli \
> --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5h****************************************************47na \
> --ssh-user opc \
> --ociprofile DEFAULT \
> --opcsshkey ~/.oci/baloise_exacc_dbaas_automation.pem
[root@dynscal-vm dynamicscaling-plugin]# /opt/dynamicscaling-plugin/dynamicscaling-plugin.bin --ocicli \
> --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5h****************************************************47na \
> --ssh-user opc \
> --ociprofile DEFAULT \
> --opcsshkey ~/.ssh/oci_exacc_vmcluster_opc \
> --nosilent
[root@dynscal-vm dynamicscaling-plugin]# 

Troubleshooting

I ran a strace.

[root@dynscal-vm dynamscaling-rpm]# strace -o Redhat_strace_DynScal_status.txt /opt/dynamicscaling/dynamicscaling.bin status

And could find following in the log file.

execve("/tmp/par-726f6f74/cache-48a43edb76f05575ffe0cb3772651416666f42b3/dynamicscaling.bin", ["/tmp/par-726f6f74/cache-48a43edb"..., "status"], 0x55edefe570f0 /* 24 vars */) = -1 EACCES (Permission denied)

This is due because the OS is hardened. There is noexec permission on /tmp. Dynamic Scaling is trying to run a file from /tmp which is forbidden.

[root@dynscal-vm ~]# findmnt -l | grep noexec | grep tmp
/tmp                            /dev/mapper/rootvg-tmp                           xfs         rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota

Solution

The solution is to setup TEMP variable to another directory : export TEMP=/var/tmp

[opc@dynscal-vm ~]$ export TEMP=/var/tmp
[opc@dynscal-vm ~]$ /opt/dynamicscaling-plugin/dynamicscaling-plugin.bin --ocicli \
> --vm-cluster-id ocid1.vmcluster.oc1.eu-zurich-1.an5h****************************************************47na \
> --ssh-user opc \
> --ociprofile DEFAULT \
> --opcsshkey ~/.ssh/oci_exacc_vmcluster_opc \
> --logpath /tmp \
> --logfile test_dynamic_scaling_plugin \
> --nosilent

│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
  Dynamicscaling remote plugin - Version: 1.0.1-13
  Copyright (c) 2021-2023 Oracle and/or its affiliates.
----------------------------------------------------------
  Author: Ruggero Citton 
  RAC Pack, Cloud Innovation and Solution Engineering Team
│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│

INFO: 2024-06-17 10:06:51: dynamicscaling_plugin log file at '/tmp/test_dynamic_scaling_plugin.log'
INFO: 2024-06-17 10:06:51: Getting exadata-infrastructure informations
INFO: 2024-06-17 10:06:52: Current OCPU: 18
INFO: 2024-06-17 10:06:52: Getting ExaCC 'public-ip' IPs list...
INFO: 2024-06-17 10:06:53: Checking ssh nodes connectivity...
INFO: 2024-06-17 10:06:54: Getting nodes load...
INFO: 2024-06-17 10:06:58: Getting cluster load
SUCCESS: 2024-06-17 10:06:58: Cluster max load is 30