{"id":15312,"date":"2020-11-29T21:53:00","date_gmt":"2020-11-29T20:53:00","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/cross-cloud-pmm-which-tcp-ports-to-open\/"},"modified":"2020-11-29T21:53:00","modified_gmt":"2020-11-29T20:53:00","slug":"cross-cloud-pmm-which-tcp-ports-to-open","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/cross-cloud-pmm-which-tcp-ports-to-open\/","title":{"rendered":"Cross-cloud PMM: which TCP ports to open"},"content":{"rendered":"

By Franck Pachot<\/h2>\n

.
\nI recently installed Percona Monitoring & Management on AWS (free tier) and here is how to monitor an instance on another cloud (OCI), in order to show which TCP port must be opened.<\/p>\n

PMM server<\/h3>\n

I installed PMM from the AWS Marketplace, following those instructions: https:\/\/www.percona.com\/doc\/percona-monitoring-and-management\/deploy\/server\/ami.html<\/a>. I’ll not reproduce the instructions, just some screenshots I took during the install:
\n\"\"<\/a>
\nI have opened the HTTPS port in order to access the console, and also configure the clients which will also connect through HTTPS (but I’m not using a signed certificate).
\n\"\"<\/a>
\nOnce installed, two targets are visible: the PMM server host (Linux) and database (PostgreSQL):
\n\"\"<\/a>
\nNote that I didn’t secure HTTPS here and I’ll have to accept insecure SSL.<\/p>\n

PMM client<\/h3>\n

I’ll monitor an Autonomous Linux instance that I have on Oracle Cloud (Free Tier). Autonomous Linux is based on OEL, which is based on RHEL (see https:\/\/www.dbi-services.com\/blog\/al7\/<\/a>) and is called “autonomous” because it updates the kernel without the need to reboot. I install the PMM Client RPM:<\/p>\n

\n[opc@al ~]$ sudo yum -y install https:\/\/repo.percona.com\/yum\/percona-release-latest.noarch.rpm\n\n\nLoaded plugins: langpacks\npercona-release-latest.noarch.rpm                                                                        |  19 kB  00:00:00\nExamining \/var\/tmp\/yum-root-YgvokG\/percona-release-latest.noarch.rpm: percona-release-1.0-25.noarch\nMarking \/var\/tmp\/yum-root-YgvokG\/percona-release-latest.noarch.rpm to be installed\nResolving Dependencies\n--> Running transaction check\n---> Package percona-release.noarch 0:1.0-25 will be installed\n--> Finished Dependency Resolution\nal7\/x86_64                                                                                               | 2.8 kB  00:00:00\nal7\/x86_64\/primary_db                                                                                    |  21 MB  00:00:00\nepel-apache-maven\/7Server\/x86_64                                                                         | 3.3 kB  00:00:00\nol7_UEKR5\/x86_64                                                                                         | 2.5 kB  00:00:00\nol7_latest\/x86_64                                                                                        | 2.7 kB  00:00:00\nol7_x86_64_userspace_ksplice                                                                             | 2.8 kB  00:00:00\n\nDependencies Resolved\n\nDependencies Resolved\n\n================================================================================================================================\n Package                        Arch                  Version               Repository                                     Size\n================================================================================================================================\nInstalling:\n percona-release                noarch                1.0-25                \/percona-release-latest.noarch                 31 k\n\nTransaction Summary\n================================================================================================================================\nInstall  1 Package\n\nTotal size: 31 k\nInstalled size: 31 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\n\nRunning transaction\n  Installing : percona-release-1.0-25.noarch                                                                                1\/1\n* Enabling the Percona Original repository\n All done!\n* Enabling the Percona Release repository\n All done!\nThe percona-release package now contains a percona-release script that can enable additional repositories for our newer products.\n\nFor example, to enable the Percona Server 8.0 repository use:\n\n  percona-release setup ps80\n\nNote: To avoid conflicts with older product versions, the percona-release setup command may disable our original repository for some products.\n\nFor more information, please visit:\n  https:\/\/www.percona.com\/doc\/percona-repo-config\/percona-release.html\n\n  Verifying  : percona-release-1.0-25.noarch                                                                                1\/1\n\nInstalled:\n  percona-release.noarch 0:1.0-25\n\n\n<\/code><\/pre>\n
This packages helps to enable additional repositories. Here, I need the PMM 2 Client:<\/p>\n
\n[opc@al ~]$ sudo percona-release enable pmm2-client\n\n\n* Enabling the PMM2 Client repository\n All done!\n\n<\/code><\/pre>\n
Once enabled, it is easy to install it with YUM:<\/p>\n
\n[opc@al ~]$ sudo yum install -y pmm2-client\n\n\nLoaded plugins: langpacks\npercona-release-noarch                                                                                   | 2.9 kB  00:00:00\npercona-release-x86_64                                                                                   | 2.9 kB  00:00:00\npmm2-client-release-x86_64                                                                               | 2.9 kB  00:00:00\nprel-release-noarch                                                                                      | 2.9 kB  00:00:00\n(1\/4): percona-release-noarch\/7Server\/primary_db                                                         |  24 kB  00:00:00\n(2\/4): pmm2-client-release-x86_64\/7Server\/primary_db                                                     | 3.5 kB  00:00:00\n(3\/4): prel-release-noarch\/7Server\/primary_db                                                            | 2.5 kB  00:00:00\n(4\/4): percona-release-x86_64\/7Server\/primary_db                                                         | 1.2 MB  00:00:00\nResolving Dependencies\n--> Running transaction check\n---> Package pmm2-client.x86_64 0:2.11.1-6.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================================================================\n Package                     Arch                   Version                        Repository                              Size\n================================================================================================================================\nInstalling:\n pmm2-client                 x86_64                 2.11.1-6.el7                   percona-release-x86_64                  42 M\n\nTransaction Summary\n================================================================================================================================\nInstall  1 Package\n\nTotal download size: 42 M\nInstalled size: 42 M\nDownloading packages:\nwarning: \/var\/cache\/yum\/x86_64\/7Server\/percona-release-x86_64\/packages\/pmm2-client-2.11.1-6.el7.x86_64.rpm: Header V4 RSA\/SHA256\n Signature, key ID 8507efa5: NOKEY\nPublic key for pmm2-client-2.11.1-6.el7.x86_64.rpm is not installed\npmm2-client-2.11.1-6.el7.x86_64.rpm                                                                      |  42 MB  00:00:07\nRetrieving key from file:\/\/\/etc\/pki\/rpm-gpg\/PERCONA-PACKAGING-KEY\nImporting GPG key 0x8507EFA5:\n Userid     : \"Percona MySQL Development Team (Packaging key) \"\n Fingerprint: 4d1b b29d 63d9 8e42 2b21 13b1 9334 a25f 8507 efa5\n Package    : percona-release-1.0-25.noarch (@\/percona-release-latest.noarch)\n From       : \/etc\/pki\/rpm-gpg\/PERCONA-PACKAGING-KEY\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n  Installing : pmm2-client-2.11.1-6.el7.x86_64                                                                              1\/1\n  Verifying  : pmm2-client-2.11.1-6.el7.x86_64                                                                              1\/1\n\nInstalled:\n  pmm2-client.x86_64 0:2.11.1-6.el7\n\nComplete!\n\n<\/code><\/pre>\n
That’s all for software installation. I just need to configure the agent to connect to the PMM Server:<\/p>\n
\n[opc@al ~]$ sudo pmm-admin config --server-url https:\/\/admin:secretpassword@18.194.119.174 --server-insecure-tls $(curl ident.me) generic OPC-$(hostname)\n\n\nChecking local pmm-agent status...\npmm-agent is running.\nRegistering pmm-agent on PMM Server...\nRegistered.\nConfiguration file \/usr\/local\/percona\/pmm2\/config\/pmm-agent.yaml updated.\nReloading pmm-agent configuration...\nConfiguration reloaded.\nChecking local pmm-agent status...\npmm-agent is running.\n<\/code><\/pre>\n
As you can see, I use the “ident-me” web service to identify my IP address, but you probably know your public IP.<\/p>\n
This configuration goes to a file, which you should protect because it contains the password in clear text:<\/p>\n
\n[opc@al ~]$ ls -l \/usr\/local\/percona\/pmm2\/config\/pmm-agent.yaml\n-rw-r-----. 1 pmm-agent pmm-agent 805 Nov 28 20:22 \/usr\/local\/percona\/pmm2\/config\/pmm-agent.yaml\n<\/code><\/pre>\n
\n# Updated by `pmm-agent setup`.\n---\nid: \/agent_id\/853027e6-563e-42b8-a417-f144541358ff\nlisten-port: 7777\nserver:\n  address: 18.194.119.174:443\n  username: admin\n  password: secretpassword\n  insecure-tls: true\npaths:\n  exporters_base: \/usr\/local\/percona\/pmm2\/exporters\n  node_exporter: \/usr\/local\/percona\/pmm2\/exporters\/node_exporter\n  mysqld_exporter: \/usr\/local\/percona\/pmm2\/exporters\/mysqld_exporter\n  mongodb_exporter: \/usr\/local\/percona\/pmm2\/exporters\/mongodb_exporter\n  postgres_exporter: \/usr\/local\/percona\/pmm2\/exporters\/postgres_exporter\n  proxysql_exporter: \/usr\/local\/percona\/pmm2\/exporters\/proxysql_exporter\n  rds_exporter: \/usr\/local\/percona\/pmm2\/exporters\/rds_exporter\n  tempdir: \/tmp\n  pt_summary: \/usr\/local\/percona\/pmm2\/tools\/pt-summary\nports:\n  min: 42000\n  max: 51999\ndebug: false\ntrace: false\n<\/code><\/pre>\n
What is interesting here is the port that is used for the server to connect to pull metrics from the client: 42000<\/p>\n
I’ll need to open this port. I can see the error from the PMM server: https:\/\/18.194.119.174\/prometheus\/targets<\/p>\n
\"\"<\/a><\/p>\n
I open this port on the host:<\/p>\n
\n[opc@al ~]$ sudo iptables -I INPUT 5 -i ens3 -p tcp --dport 42000 -m state --state NEW,ESTABLISHED -j ACCEPT\n<\/code><\/pre>\n
and on the ingress rules as well:
\n\"\"<\/a><\/p>\n
Testing<\/h3>\n
I’m running two processes here to test if I get the right metrics<\/p>\n
\n[opc@al ~]$ while true ; do sudo dd bs=100M count=1                if=$(df -Th | sort -rhk3 | awk '\/^[\/]dev\/{print $1;exit}') of=\/dev\/null ; done &\n[opc@al ~]$ while true ; do sudo dd bs=100M count=10G iflag=direct if=$(df -Th | sort -rhk3 | awk '\/^[\/]dev\/{print $1;exit}') of=\/dev\/null ; done &\n<\/code><\/pre>\n
The latter will do mostly I\/O as I read with O_DIRECT and the former mainly system CPU as it reads from filesystem cache<\/p>\n
Here is the Grafana dashboard from PMM:
\n\"\"<\/a>
\nI see my two processes, and 80% of CPU stolen by the hypervisor as I’m running on the Free Tier here which provides 1\/8th of OCPU.<\/p>\n
If you have MySQL or PostgreSQL databases there, they can easily be monitored (“pmm-admin add MySQL” or “pmm-admin add MySQL” you can see all that in Elisa Usai demo: https:\/\/youtu.be\/VgOR_GCUpVw?t=1558<\/a>).<\/p>\n
Last test, let’s see what happens if the monitored host reboots:<\/h3>\n
\n[opc@al ~]$ date\nSat Nov 28 22:54:36 CET 2020\n[opc@al ~]$ uptrack-uname -a\nLinux al 4.14.35-2025.402.2.1.el7uek.x86_64 #2 SMP Fri Oct 23 22:27:16 PDT 2020 x86_64 x86_64 x86_64 GNU\/Linux\n[opc@al ~]$ uname -a\nLinux al 4.14.35-1902.301.1.el7uek.x86_64 #2 SMP Tue Mar 31 16:50:32 PDT 2020 x86_64 x86_64 x86_64 GNU\/Linux\n[opc@al ~]$ sudo systemctl reboot\nConnection to 130.61.159.88 closed by remote host.\nConnection to 130.61.159.88 closed.\n<\/code><\/pre>\n
Yes… I do not reboot it frequently because it is Autonomous Linux and the Effective kernel is up to date (latest patches from October) even if the last restart was in March. But this deserves a test.<\/p>\n
The first interesting thing is that PMM seems to keep the last read metrics for a while:
\n\"\"<\/a>
\nThe host was shut down at 22:55 and it shows the last metrics for 5 minutes before stopping.<\/p>\n
I had to wait for a while because my Availability Domain was out of capacity for the free tier:<\/p>\n
\n
Sometimes, even an Autonomous Linux must be rebooted. And if you are unlucky, there are no available servers in the Availability Domain and it remains stopped\ud83e\udd37\u200d\u2642\ufe0f
(this is Free Tier) pic.twitter.com\/sd3KIeoHkd<\/a><\/p>\n
— Franck Pachot (@FranckPachot) November 28, 2020<\/a><\/p><\/blockquote>\n