{"id":32057,"date":"2024-05-14T09:40:58","date_gmt":"2024-05-14T07:40:58","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/?p=32057"},"modified":"2024-05-14T09:41:01","modified_gmt":"2024-05-14T07:41:01","slug":"upgrade-etcd-in-a-patroni-cluster","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/","title":{"rendered":"Upgrade etcd in a patroni cluster"},"content":{"rendered":"\n<p>In a distributed database system like PostgreSQL managed by Patroni, etcd plays a critical role as the distributed key-value store for cluster coordination and configuration. As your system evolves, upgrading etcd becomes necessary to leverage new features, bug fixes, and security enhancements. However, upgrading etcd in a live cluster requires careful planning and execution to ensure data integrity. In this guide, we&#8217;ll walk through the process of upgrading etcd from version 3.4.25 to 3.5.12 in a Patroni cluster, based on the detailed notes I took during the upgrade process.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-check-the-upgrade-checklist\">Check the upgrade checklist<\/h2>\n\n\n\n<p>Before trying to upgrade, it is important to have a look at all the deprecated features and at the upgrade requirements. In our case, to upgrade to version 3.5.x, it is mandatory that the running cluster is healthy and at least in version 3.4 already.<br>You can find all this information on the official etcd documentation:<br><a href=\"https:\/\/etcd.io\/docs\/v3.3\/upgrades\/upgrade_3_5\/\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/etcd.io\/docs\/v3.3\/upgrades\/upgrade_3_5\/<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-no-downtime-for-patroni\">No downtime for patroni<\/h2>\n\n\n\n<p>During the upgrade process, an etcd cluster can accommodate a mix of etcd member versions, functioning based on the protocol of the lowest common version present. The cluster achieves the upgraded status only when all its members are updated to version 3.5. Internally, etcd members negotiate among themselves to establish the overall cluster version, influencing the reported version and the features supported by the cluster.<\/p>\n\n\n\n<p>In most scenarios, transitioning from etcd 3.4 to 3.5 can be accomplished seamlessly through a rolling upgrade process, ensuring zero downtime. Sequentially halt the etcd v3.4 processes, substituting them with etcd v3.5 processes. Upon completion of the migration to v3.5 across all nodes, the enhanced functionalities introduced in v3.5 become accessible to the cluster.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-preparing-for-the-upgrade\">Preparing for the Upgrade<\/h2>\n\n\n\n<p>Before starting the upgrade process, it&#8217;s essential to make adequate preparations to minimize any potential risks or disruptions. Here are some preliminary steps:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Check current etcd version<\/strong><\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#091;pgt001] postgres@patroni-1:\/postgres\/app\/postgres&gt; etcdctl version\netcdctl version: 3.4.25\nAPI version: 3.4<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Backup etcd data<\/strong><\/li>\n<\/ul>\n\n\n\n<p>Use <code>etcdctl<\/code> to create a snapshot of the etcd data. This ensures that you have a fallback option in case something goes wrong during the upgrade process.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#091;pgt001] postgres@patroni-1:~&gt; etcdctl snapshot save backup.db\n{\"level\":\"info\",\"ts\":1710507460.523724,\"caller\":\"snapshot\/v3_snapshot.go:119\",\"msg\":\"created temporary db file\",\"path\":\"backup.db.part\"}\n{\"level\":\"info\",\"ts\":\"2024-03-15T13:57:40.538461+0100\",\"caller\":\"clientv3\/maintenance.go:200\",\"msg\":\"opened snapshot stream; downloading\"}\n{\"level\":\"info\",\"ts\":1710507460.539052,\"caller\":\"snapshot\/v3_snapshot.go:127\",\"msg\":\"fetching snapshot\",\"endpoint\":\"127.0.0.1:2379\"}\n{\"level\":\"info\",\"ts\":\"2024-03-15T13:57:40.548342+0100\",\"caller\":\"clientv3\/maintenance.go:208\",\"msg\":\"completed snapshot read; closing\"}\n{\"level\":\"info\",\"ts\":1710507460.5576544,\"caller\":\"snapshot\/v3_snapshot.go:142\",\"msg\":\"fetched snapshot\",\"endpoint\":\"127.0.0.1:2379\",\"size\":\"57 kB\",\"took\":0.030259485}\n{\"level\":\"info\",\"ts\":1710507460.5580025,\"caller\":\"snapshot\/v3_snapshot.go:152\",\"msg\":\"saved\",\"path\":\"backup.db\"}\nSnapshot saved at backup.db\n&#091;pgt001] postgres@patroni-1:~&gt; ll\ntotal 60\n-rw------- 1 postgres postgres 57376 Mar 15 13:57 backup.db\n\n&#091;pgt001] postgres@patroni-1:~&gt; etcdctl --write-out=table snapshot status backup.db\n+----------+----------+------------+------------+\n|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |\n+----------+----------+------------+------------+\n| 29c96081 |      107 |        117 |      57 kB |\n+----------+----------+------------+------------+<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Pause Cluster Management<\/strong><\/li>\n<\/ul>\n\n\n\n<p>Use Patroni&#8217;s <code>patronictl<\/code> to pause cluster management. This prevents any automated failover or configuration changes during the upgrade process. (<a href=\"https:\/\/patroni.readthedocs.io\/en\/latest\/pause.html\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/patroni.readthedocs.io\/en\/latest\/pause.html<\/a>)<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#091;pgt001] postgres@patroni-1:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; patronictl pause --wait\n'pause' request sent, waiting until it is recognized by all nodes\nSuccess: cluster management is paused<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-performing-the-upgrade\">Performing the Upgrade<\/h2>\n\n\n\n<p>Now that you&#8217;ve prepared your cluster for the upgrade, you can proceed with the actual upgrade steps. All the steps are performed node by node, as mentioned earlier. I will start the upgrade on the third node of my cluster, patroni-3.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Stop etcd<\/strong><\/li>\n<\/ul>\n\n\n\n<p>Stop the etcd service. This ensures that no changes are made to the cluster while the upgrade is in progress.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#091;pg133] postgres@patroni-3:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; sudo systemctl stop etcd<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Extract and Install New etcd Version<\/strong><\/li>\n<\/ul>\n\n\n\n<p>Download the new etcd binary and extract it. Then, replace the existing etcd binaries with the new ones.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#091;pg133] postgres@patroni-3:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; tar axf etcd-v3.5.12-linux-amd64.tar.gz\n&#091;pg133] postgres@patroni-3:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; mv etcd-v3.5.12-linux-amd64\/etcd* \/postgres\/app\/postgres\/local\/dmk\/bin\/\n&#091;pg133] postgres@patroni-3:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; etcdctl version\netcdctl version: 3.5.12\nAPI version: 3.5<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Start etcd<\/strong><\/li>\n<\/ul>\n\n\n\n<p>Start the upgraded etcd service<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#091;pg133] postgres@patroni-3:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; sudo systemctl start etcd\n&#091;pg133] postgres@patroni-3:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; sudo systemctl status etcd\n\u25cf etcd.service - dbi services etcd service\n     Loaded: loaded (\/etc\/systemd\/system\/etcd.service; enabled; preset: enabled)\n     Active: active (running) since Fri 2024-03-15 14:02:39 CET; 10s ago\n   Main PID: 1561 (etcd)\n      Tasks: 9 (limit: 9454)\n     Memory: 13.1M\n        CPU: 369ms\n     CGroup: \/system.slice\/etcd.service\n             \u2514\u25001561 \/postgres\/app\/postgres\/local\/dmk\/bin\/etcd --config-file \/postgres\/app\/postgres\/local\/dmk\/etc\/etcd.conf\n\nMar 15 14:02:38 patroni-3 etcd&#091;1561]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:02:38.292751+0100\",\"caller\":\"etcdserver\/server.go:783\",\"msg\":\"initialized peer connections; fast-forwarding electi&gt;\nMar 15 14:02:39 patroni-3 etcd&#091;1561]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:02:39.282054+0100\",\"logger\":\"raft\",\"caller\":\"etcdserver\/zap_raft.go:77\",\"msg\":\"raft.node: f1457fc5460d0329 elected&gt;\nMar 15 14:02:39 patroni-3 etcd&#091;1561]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:02:39.302529+0100\",\"caller\":\"etcdserver\/server.go:2068\",\"msg\":\"published local member to cluster through raft\",\"lo&gt;\nMar 15 14:02:39 patroni-3 etcd&#091;1561]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:02:39.302985+0100\",\"caller\":\"embed\/serve.go:103\",\"msg\":\"ready to serve client requests\"}\nMar 15 14:02:39 patroni-3 etcd&#091;1561]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:02:39.30307+0100\",\"caller\":\"embed\/serve.go:103\",\"msg\":\"ready to serve client requests\"}\nMar 15 14:02:39 patroni-3 etcd&#091;1561]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:02:39.302942+0100\",\"caller\":\"etcdmain\/main.go:44\",\"msg\":\"notifying init daemon\"}\nMar 15 14:02:39 patroni-3 etcd&#091;1561]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:02:39.303671+0100\",\"caller\":\"etcdmain\/main.go:50\",\"msg\":\"successfully notified init daemon\"}\nMar 15 14:02:39 patroni-3 systemd&#091;1]: Started etcd.service - dbi services etcd service.\nMar 15 14:02:39 patroni-3 etcd&#091;1561]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:02:39.304964+0100\",\"caller\":\"embed\/serve.go:187\",\"msg\":\"serving client traffic insecurely; this is strongly discou&gt;\nMar 15 14:02:39 patroni-3 etcd&#091;1561]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:02:39.305719+0100\",\"caller\":\"embed\/serve.go:187\",\"msg\":\"serving client traffic insecurely; this is strongly discou&gt;\n<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-verifying-cluster-health\">Verifying Cluster Health<\/h2>\n\n\n\n<p>After each etcd upgrade, it&#8217;s always nice to verify the health and functionality of the etcd and Patroni cluster. You can notice from the etcdtcl command that the version was upgraded on the third node.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#091;pgt001] postgres@patroni-1:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; etcdctl endpoint status --cluster -w table\n+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+\n|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |\n+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+\n| http:\/\/192.168.56.123:2379 | 90015c533cbf2e84 |  3.4.25 |   61 kB |     false |      false |        15 |        150 |                150 |        |\n| http:\/\/192.168.56.124:2379 | 9fe85e3cebf257e3 |  3.4.25 |   61 kB |     false |      false |        15 |        150 |                150 |        |\n| http:\/\/192.168.56.125:2379 | f1457fc5460d0329 |  3.5.12 |   61 kB |      true |      false |        15 |        150 |                150 |        |\n+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>&#091;pgt001] postgres@patroni-1:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; patronictl list\n+ Cluster: pgt001 (7346518467491201916) ----------+----+-----------+\n| Member   | Host           | Role    | State     | TL | Lag in MB |\n+----------+----------------+---------+-----------+----+-----------+\n| pgt001_1 | 192.168.56.123 | Leader  | running   |  5 |           |\n| pgt001_2 | 192.168.56.124 | Replica | streaming |  5 |         0 |\n+----------+----------------+---------+-----------+----+-----------+\n Maintenance mode: on<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-continuing-the-installation-on-the-other-nodes\">Continuing the installation on the other nodes<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Upgrading etcd on the second node of the cluster<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#091;pgt001] postgres@patroni-2:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; sudo systemctl stop etcd\n&#091;pgt001] postgres@patroni-2:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; tar axf etcd-v3.5.12-linux-amd64.tar.gz\n&#091;pgt001] postgres@patroni-2:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; mv etcd-v3.5.12-linux-amd64\/etcd* \/postgres\/app\/postgres\/local\/dmk\/bin\/\n&#091;pgt001] postgres@patroni-2:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; etcdctl version\netcdctl version: 3.5.12\nAPI version: 3.5\n&#091;pgt001] postgres@patroni-2:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; sudo systemctl start etcd\n&#091;pgt001] postgres@patroni-2:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; sudo systemctl status etcd\n\u25cf etcd.service - dbi services etcd service\n     Loaded: loaded (\/etc\/systemd\/system\/etcd.service; enabled; preset: enabled)\n     Active: active (running) since Fri 2024-03-15 14:04:46 CET; 4s ago\n   Main PID: 1791 (etcd)\n      Tasks: 7 (limit: 9454)\n     Memory: 9.7M\n        CPU: 295ms\n     CGroup: \/system.slice\/etcd.service\n             \u2514\u25001791 \/postgres\/app\/postgres\/local\/dmk\/bin\/etcd --config-file \/postgres\/app\/postgres\/local\/dmk\/etc\/etcd.conf\n\nMar 15 14:04:45 patroni-2 etcd&#091;1791]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:04:45.690431+0100\",\"caller\":\"rafthttp\/stream.go:274\",\"msg\":\"established TCP streaming connection with remote peer\"&gt;\nMar 15 14:04:46 patroni-2 etcd&#091;1791]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:04:46.739502+0100\",\"logger\":\"raft\",\"caller\":\"etcdserver\/zap_raft.go:77\",\"msg\":\"raft.node: 9fe85e3cebf257e3 elected&gt;\nMar 15 14:04:46 patroni-2 etcd&#091;1791]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:04:46.75204+0100\",\"caller\":\"etcdserver\/server.go:2068\",\"msg\":\"published local member to cluster through raft\",\"loc&gt;\nMar 15 14:04:46 patroni-2 etcd&#091;1791]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:04:46.752889+0100\",\"caller\":\"embed\/serve.go:103\",\"msg\":\"ready to serve client requests\"}\nMar 15 14:04:46 patroni-2 etcd&#091;1791]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:04:46.753543+0100\",\"caller\":\"etcdmain\/main.go:44\",\"msg\":\"notifying init daemon\"}\nMar 15 14:04:46 patroni-2 systemd&#091;1]: Started etcd.service - dbi services etcd service.\nMar 15 14:04:46 patroni-2 etcd&#091;1791]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:04:46.754213+0100\",\"caller\":\"embed\/serve.go:187\",\"msg\":\"serving client traffic insecurely; this is strongly discou&gt;\nMar 15 14:04:46 patroni-2 etcd&#091;1791]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:04:46.757187+0100\",\"caller\":\"embed\/serve.go:103\",\"msg\":\"ready to serve client requests\"}\nMar 15 14:04:46 patroni-2 etcd&#091;1791]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:04:46.757933+0100\",\"caller\":\"embed\/serve.go:187\",\"msg\":\"serving client traffic insecurely; this is strongly discou&gt;\nMar 15 14:04:46 patroni-2 etcd&#091;1791]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:04:46.75994+0100\",\"caller\":\"etcdmain\/main.go:50\",\"msg\":\"successfully notified init daemon\"}\n\n\n&#091;pgt001] postgres@patroni-2:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; patronictl list\n+ Cluster: pgt001 (7346518467491201916) ----------+----+-----------+\n| Member   | Host           | Role    | State     | TL | Lag in MB |\n+----------+----------------+---------+-----------+----+-----------+\n| pgt001_1 | 192.168.56.123 | Leader  | running   |  5 |           |\n| pgt001_2 | 192.168.56.124 | Replica | streaming |  5 |         0 |\n+----------+----------------+---------+-----------+----+-----------+\n Maintenance mode: on\n<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Upgrading etcd on the third node of the cluster<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>&#091;pgt001] postgres@patroni-1:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; sudo systemctl stop etcd<\/code><\/pre>\n\n\n\n<p>Now that we stopped etcd on the server where is our Patroni leader node, let&#8217;s take a look at our patroni cluster status.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#091;pgt001] postgres@patroni-2:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; patronictl list\n2024-03-15 14:05:52,778 - ERROR - Failed to get list of machines from http:\/\/192.168.56.123:2379\/v3beta: MaxRetryError(\"HTTPConnectionPool(host='192.168.56.123', port=2379): Max retries exceeded with url: \/version (Caused by NewConnectionError('&lt;urllib3.connection.HTTPConnection object at 0x7f3584365590&gt;: Failed to establish a new connection: &#091;Errno 111] Connection refused'))\")\n+ Cluster: pgt001 (7346518467491201916) ----------+----+-----------+\n| Member   | Host           | Role    | State     | TL | Lag in MB |\n+----------+----------------+---------+-----------+----+-----------+\n| pgt001_1 | 192.168.56.123 | Leader  | running   |  5 |           |\n| pgt001_2 | 192.168.56.124 | Replica | streaming |  5 |         0 |\n+----------+----------------+---------+-----------+----+-----------+\n Maintenance mode: on\n&#091;pgt001] postgres@patroni-2:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; sq\npsql (14.7 dbi services build)\nType \"help\" for help.\n\npostgres=# exit<\/code><\/pre>\n\n\n\n<p>We can notice that our Patroni cluster is still up and running and that PostgreSQL cluster is still reachable. Also, thanks to patroni maintenance mode, no failover or configuration changes are happening.<\/p>\n\n\n\n<p>Let&#8217;s continue with the installation<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#091;pgt001] postgres@patroni-1:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; tar axf etcd-v3.5.12-linux- amd64.tar.gz\n&#091;pgt001] postgres@patroni-1:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; mv etcd-v3.5.12-linux-amd64 \/etcd* \/postgres\/app\/postgres\/local\/dmk\/bin\/\n&#091;pgt001] postgres@patroni-1:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; etcdctl version\netcdctl version: 3.5.12\nAPI version: 3.5\n&#091;pgt001] postgres@patroni-1:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; sudo systemctl start etcd\n&#091;pgt001] postgres@patroni-1:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; sudo systemctl status etcd\n\u25cf etcd.service - dbi services etcd service\nLoaded: loaded (\/etc\/systemd\/system\/etcd.service; enabled; preset: enabled)\nActive: active (running) since Fri 2024-03-15 14:07:12 CET; 3s ago\nMain PID: 1914 (etcd)\nTasks: 7 (limit: 9454)\nMemory: 15.9M\nCPU: 160ms\nCGroup: \/system.slice\/etcd.service\n\u2514\u25001914 \/postgres\/app\/postgres\/local\/dmk\/bin\/etcd --config-file \/postgres\/app\/postgres\/local\/dmk\/etc\/etcd.conf\n\nMar 15 14:07:12 patroni-1 etcd&#091;1914]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:07:12.180191+0100\",\"caller\":\"etcdserver\/server.go:2068\",\"msg\":\"published local member to cluster through raft\",\"lo&gt;\nMar 15 14:07:12 patroni-1 etcd&#091;1914]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:07:12.180266+0100\",\"caller\":\"embed\/serve.go:103\",\"msg\":\"ready to serve client requests\"}\nMar 15 14:07:12 patroni-1 etcd&#091;1914]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:07:12.181162+0100\",\"caller\":\"embed\/serve.go:103\",\"msg\":\"ready to serve client requests\"}\nMar 15 14:07:12 patroni-1 etcd&#091;1914]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:07:12.182377+0100\",\"caller\":\"embed\/serve.go:187\",\"msg\":\"serving client traffic insecurely; this is strongly discou&gt;\nMar 15 14:07:12 patroni-1 etcd&#091;1914]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:07:12.182625+0100\",\"caller\":\"embed\/serve.go:187\",\"msg\":\"serving client traffic insecurely; this is strongly discou&gt;\nMar 15 14:07:12 patroni-1 etcd&#091;1914]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:07:12.183861+0100\",\"caller\":\"etcdmain\/main.go:44\",\"msg\":\"notifying init daemon\"}\nMar 15 14:07:12 patroni-1 systemd&#091;1]: Started etcd.service - dbi services etcd service.\nMar 15 14:07:12 patroni-1 etcd&#091;1914]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:07:12.187771+0100\",\"caller\":\"etcdmain\/main.go:50\",\"msg\":\"successfully notified init daemon\"}\nMar 15 14:07:12 patroni-1 etcd&#091;1914]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:07:12.195369+0100\",\"caller\":\"membership\/cluster.go:576\",\"msg\":\"updated cluster version\",\"cluster-id\":\"571a53e78674&gt;\nMar 15 14:07:12 patroni-1 etcd&#091;1914]: {\"level\":\"info\",\"ts\":\"2024-03-15T14:07:12.195541+0100\",\"caller\":\"api\/capability.go:75\",\"msg\":\"enabled capabilities for version\",\"cluster-version\":\"3.5&gt;<\/code><\/pre>\n\n\n\n<p>We now have upgraded etcd on all our nodes and we need to control the status of our clusters.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#091;pgt001] postgres@patroni-1:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; patronictl list\n+ Cluster: pgt001 (7346518467491201916) ----------+----+-----------+\n| Member   | Host           | Role    | State     | TL | Lag in MB |\n+----------+----------------+---------+-----------+----+-----------+\n| pgt001_1 | 192.168.56.123 | Leader  | running   |  5 |           |\n| pgt001_2 | 192.168.56.124 | Replica | streaming |  5 |         0 |\n+----------+----------------+---------+-----------+----+-----------+\n\n&#091;pgt001] postgres@patroni-1:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; etcdctl endpoint status --cluster -w table\n+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+\n|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |\n+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+\n| http:\/\/192.168.56.123:2379 | 90015c533cbf2e84 |  3.5.12 |   61 kB |     false |      false |        15 |        150 |                150 |        |\n| http:\/\/192.168.56.124:2379 | 9fe85e3cebf257e3 |  3.5.12 |   61 kB |     false |      false |        15 |        150 |                150 |        |\n| http:\/\/192.168.56.125:2379 | f1457fc5460d0329 |  3.5.12 |   61 kB |      true |      false |        15 |        150 |                150 |        |\n+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+\n\n&#091;pgt001] postgres@patroni-1:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; etcdctl version\netcdctl version: 3.5.12\nAPI version: 3.5<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-resume-cluster-management\"><strong>Resume Cluster Management<\/strong><\/h2>\n\n\n\n<p>Once you&#8217;ve confirmed that the upgrade was successful, resume cluster management to allow Patroni to resume its normal operations and quit maintenance mode.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#091;pgt001] postgres@patroni-1:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; patronictl resume --wait\n'resume' request sent, waiting until it is recognized by all nodes\nSuccess: cluster management is resumed\n&#091;pgt001] postgres@patroni-1:\/postgres\/app\/postgres\/local\/dmk\/bin&gt; patronictl list\n+ Cluster: pgt001 (7346518467491201916) ----------+----+-----------+\n| Member   | Host           | Role    | State     | TL | Lag in MB |\n+----------+----------------+---------+-----------+----+-----------+\n| pgt001_1 | 192.168.56.123 | Leader  | running   |  5 |           |\n| pgt001_2 | 192.168.56.124 | Replica | streaming |  5 |         0 |\n+----------+----------------+---------+-----------+----+-----------+<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-conclusion\">Conclusion<\/h2>\n\n\n\n<p>Upgrading etcd in a Patroni cluster is a critical maintenance task that requires careful planning and execution. By following the steps outlined in this guide and leveraging the detailed notes taken during the upgrade process, you can ensure a smooth and successful upgrade while ensuring data integrity. Remember to always test the upgrade process in a staging environment before performing it in production to mitigate any potential risks.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In a distributed database system like PostgreSQL managed by Patroni, etcd plays a critical role as the distributed key-value store for cluster coordination and configuration. As your system evolves, upgrading etcd becomes necessary to leverage new features, bug fixes, and security enhancements. However, upgrading etcd in a live cluster requires careful planning and execution to [&hellip;]<\/p>\n","protected":false},"author":87,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[229,83],"tags":[38,1618,1543,77,2601],"type_dbi":[3307,3060,2749,3308],"class_list":["post-32057","post","type-post","status-publish","format-standard","hentry","category-database-administration-monitoring","category-postgresql","tag-cluster","tag-etcd","tag-patroni","tag-postgresql","tag-upgrade-2","type-etcd","type-patroni","type-postgresql","type-upgrade"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.2 (Yoast SEO v27.2) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Upgrade etcd in a patroni cluster - dbi Blog<\/title>\n<meta name=\"description\" content=\"Learn how to seamlessly upgrade etcd from version 3.4.25 to 3.5.12 in a Patroni cluster, ensuring zero downtime and maintaining data integrity. Follow these detailed steps for a smooth transition and enhanced cluster functionality.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Upgrade etcd in a patroni cluster\" \/>\n<meta property=\"og:description\" content=\"Learn how to seamlessly upgrade etcd from version 3.4.25 to 3.5.12 in a Patroni cluster, ensuring zero downtime and maintaining data integrity. Follow these detailed steps for a smooth transition and enhanced cluster functionality.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2024-05-14T07:40:58+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-05-14T07:41:01+00:00\" \/>\n<meta name=\"author\" content=\"Joan Frey\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Joan Frey\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/\"},\"author\":{\"name\":\"Joan Frey\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/c03c47649664fe73b27ce457e99f5b06\"},\"headline\":\"Upgrade etcd in a patroni cluster\",\"datePublished\":\"2024-05-14T07:40:58+00:00\",\"dateModified\":\"2024-05-14T07:41:01+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/\"},\"wordCount\":710,\"commentCount\":0,\"keywords\":[\"Cluster\",\"etcd\",\"Patroni\",\"PostgreSQL\",\"upgrade\"],\"articleSection\":[\"Database Administration &amp; Monitoring\",\"PostgreSQL\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/\",\"name\":\"Upgrade etcd in a patroni cluster - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"datePublished\":\"2024-05-14T07:40:58+00:00\",\"dateModified\":\"2024-05-14T07:41:01+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/c03c47649664fe73b27ce457e99f5b06\"},\"description\":\"Learn how to seamlessly upgrade etcd from version 3.4.25 to 3.5.12 in a Patroni cluster, ensuring zero downtime and maintaining data integrity. Follow these detailed steps for a smooth transition and enhanced cluster functionality.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Upgrade etcd in a patroni cluster\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/c03c47649664fe73b27ce457e99f5b06\",\"name\":\"Joan Frey\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/1e650cf665b4d44dd186355827c0b049d2f95c8cbb45fd10d4e7cb255be67ecb?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/1e650cf665b4d44dd186355827c0b049d2f95c8cbb45fd10d4e7cb255be67ecb?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/1e650cf665b4d44dd186355827c0b049d2f95c8cbb45fd10d4e7cb255be67ecb?s=96&d=mm&r=g\",\"caption\":\"Joan Frey\"},\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/joanfrey\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Upgrade etcd in a patroni cluster - dbi Blog","description":"Learn how to seamlessly upgrade etcd from version 3.4.25 to 3.5.12 in a Patroni cluster, ensuring zero downtime and maintaining data integrity. Follow these detailed steps for a smooth transition and enhanced cluster functionality.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/","og_locale":"en_US","og_type":"article","og_title":"Upgrade etcd in a patroni cluster","og_description":"Learn how to seamlessly upgrade etcd from version 3.4.25 to 3.5.12 in a Patroni cluster, ensuring zero downtime and maintaining data integrity. Follow these detailed steps for a smooth transition and enhanced cluster functionality.","og_url":"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/","og_site_name":"dbi Blog","article_published_time":"2024-05-14T07:40:58+00:00","article_modified_time":"2024-05-14T07:41:01+00:00","author":"Joan Frey","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Joan Frey","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/"},"author":{"name":"Joan Frey","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/c03c47649664fe73b27ce457e99f5b06"},"headline":"Upgrade etcd in a patroni cluster","datePublished":"2024-05-14T07:40:58+00:00","dateModified":"2024-05-14T07:41:01+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/"},"wordCount":710,"commentCount":0,"keywords":["Cluster","etcd","Patroni","PostgreSQL","upgrade"],"articleSection":["Database Administration &amp; Monitoring","PostgreSQL"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/","url":"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/","name":"Upgrade etcd in a patroni cluster - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2024-05-14T07:40:58+00:00","dateModified":"2024-05-14T07:41:01+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/c03c47649664fe73b27ce457e99f5b06"},"description":"Learn how to seamlessly upgrade etcd from version 3.4.25 to 3.5.12 in a Patroni cluster, ensuring zero downtime and maintaining data integrity. Follow these detailed steps for a smooth transition and enhanced cluster functionality.","breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/upgrade-etcd-in-a-patroni-cluster\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Upgrade etcd in a patroni cluster"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/c03c47649664fe73b27ce457e99f5b06","name":"Joan Frey","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1e650cf665b4d44dd186355827c0b049d2f95c8cbb45fd10d4e7cb255be67ecb?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1e650cf665b4d44dd186355827c0b049d2f95c8cbb45fd10d4e7cb255be67ecb?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1e650cf665b4d44dd186355827c0b049d2f95c8cbb45fd10d4e7cb255be67ecb?s=96&d=mm&r=g","caption":"Joan Frey"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/joanfrey\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/32057","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/87"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=32057"}],"version-history":[{"count":23,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/32057\/revisions"}],"predecessor-version":[{"id":32167,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/32057\/revisions\/32167"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=32057"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=32057"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=32057"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=32057"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}