{"id":12975,"date":"2019-11-06T10:43:38","date_gmt":"2019-11-06T09:43:38","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/"},"modified":"2024-09-10T17:37:44","modified_gmt":"2024-09-10T15:37:44","slug":"how-to-scale-up-a-patroni-cluster","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/","title":{"rendered":"How to scale up a Patroni cluster"},"content":{"rendered":"

During the preparation of my presentation for the pgconf.eu I ran into one big issue. I had to stop my cluster to add a new node. That was not the way I wanted to archive this. I want a high availability solution, that can be scaled up without any outage. Due to a little hint during the pgconf.eu I was able to find a solution. In this post I will show the manually scale up, without using a playbook.
\n<\/p>\n

Starting position<\/h3>\n

We start with a 3 node patroni cluster which can be created using this blog post<\/a>.
\nNow we want to add a fourth node to the existing etcd and Patroni cluster. In case you also need a playbook to install a forth node, check out my GitHub repository<\/a>. <\/p>\n

Scale up the etcd cluster <\/h3>\n

This step is only needed, when you want to scale up your etcd cluster as well. To scale up a Patroni cluster it is not necessary to scale up etcd cluster. You can, of course, scale up Patroni without adding more etcd cluster members. But maybe someone also needs to scale up his etcd cluster and searches for a solution. If not, just jump to the next step. <\/p>\n

Be sure the etcd and patroni service are not started on the forth node.<\/p>\n

\npostgres@patroni4:\/home\/postgres\/ [PG1] systemctl status etcd\n\u25cf etcd.service - dbi services etcd service\n   Loaded: loaded (\/etc\/systemd\/system\/etcd.service; enabled; vendor preset: disabled)\n   Active: inactive (dead)\npostgres@patroni4:\/home\/postgres\/ [PG1] systemctl status patroni\n\u25cf patroni.service - dbi services patroni service\n   Loaded: loaded (\/etc\/systemd\/system\/patroni.service; enabled; vendor preset: disabled)\n   Active: inactive (dead)\npostgres@patroni4:\/home\/postgres\/ [PG1]\n<\/pre>\n
Make the following adjustments in the etcd.conf of the 4th node.<\/p>\n
\npostgres@patroni4:\/home\/postgres\/ [PG1] cat \/u01\/app\/postgres\/local\/dmk\/etc\/etcd.conf\nname: patroni4\ndata-dir: \/u02\/pgdata\/etcd\ninitial-advertise-peer-urls: http:\/\/192.168.22.114:2380\nlisten-peer-urls: http:\/\/192.168.22.114:2380\nlisten-client-urls: http:\/\/192.168.22.114:2379,http:\/\/localhost:2379\nadvertise-client-urls: http:\/\/192.168.22.114:2379\ninitial-cluster-state: 'existing'\ninitial-cluster: patroni1=http:\/\/192.168.22.111:2380,patroni2=http:\/\/192.168.22.112:2380,patroni3=http:\/\/192.168.22.113:2380,patroni4=http:\/\/192.168.22.114:2380\n<\/pre>\n
Next add the new etcd member to the existing etcd cluster. You can execute that on every existing member of the cluster.<\/p>\n
\npostgres@patroni1:\/home\/postgres\/ [PG1] etcdctl member add patroni4 http:\/\/192.168.22.114:2380\nAdded member named patroni4 with ID dd9fab8349b3cfc to cluster\n\nETCD_NAME=\"patroni4\"\nETCD_INITIAL_CLUSTER=\"patroni4=http:\/\/192.168.22.114:2380,patroni1=http:\/\/192.168.22.111:2380,patroni2=http:\/\/192.168.22.112:2380,patroni3=http:\/\/192.168.22.113:2380\"\nETCD_INITIAL_CLUSTER_STATE=\"existing\"\n<\/pre>\n
Now you can start the etcd service on the 4th node.<\/p>\n
\npostgres@patroni4:\/home\/postgres\/ [PG1] sudo systemctl start etcd\npostgres@patroni4:\/home\/postgres\/ [PG1] systemctl status etcd\n\u25cf etcd.service - dbi services etcd service\n   Loaded: loaded (\/etc\/systemd\/system\/etcd.service; enabled; vendor preset: disabled)\n   Active: active (running) since Thu 2019-10-17 16:39:16 CEST; 9s ago\n Main PID: 8239 (etcd)\n   CGroup: \/system.slice\/etcd.service\n           \u2514\u25008239 \/u01\/app\/postgres\/local\/dmk\/bin\/etcd --config-file \/u01\/app\/postgres\/local\/dmk\/etc\/etcd.conf\npostgres@patroni4:\/home\/postgres\/ [PG1]\n<\/pre>\n
And after a short check, we can see, that Node 4 is added to the existing cluster<\/p>\n
\npostgres@patroni4:\/home\/postgres\/ [PG1] etcdctl member list\ndd9fab8349b3cfc: name=patroni4 peerURLs=http:\/\/192.168.22.114:2380 clientURLs=http:\/\/192.168.22.114:2379 isLeader=false\n16e1dca5ee237693: name=patroni1 peerURLs=http:\/\/192.168.22.111:2380 clientURLs=http:\/\/192.168.22.111:2379 isLeader=false\n28a43bb36c801ed4: name=patroni2 peerURLs=http:\/\/192.168.22.112:2380 clientURLs=http:\/\/192.168.22.112:2379 isLeader=false\n5ba7b55764fad76e: name=patroni3 peerURLs=http:\/\/192.168.22.113:2380 clientURLs=http:\/\/192.168.22.113:2379 isLeader=true\n<\/pre>\n
 Scale up Patroni<\/h3>\n
Scale up the Patroni cluster is also really easy.
\nAdjust the host entry in the patroni.yml on the new node.<\/p>\n
\npostgres@patroni4:\/home\/postgres\/ [PG1] cat \/u01\/app\/postgres\/local\/dmk\/etc\/patroni.yml | grep hosts\n  hosts: 192.168.22.111:2379,192.168.22.112:2379,192.168.22.113:2379,192.168.22.114:2379\n<\/pre>\n
Afterwards, start the Patroni service.<\/p>\n
\npostgres@patroni4:\/home\/postgres\/ [PG1] sudo systemctl start patroni\npostgres@patroni4:\/home\/postgres\/ [PG1] systemctl status patroni\n\u25cf patroni.service - dbi services patroni service\n   Loaded: loaded (\/etc\/systemd\/system\/patroni.service; enabled; vendor preset: disabled)\n   Active: active (running) since Thu 2019-10-17 17:03:19 CEST; 5s ago\n  Process: 8476 ExecStartPre=\/usr\/bin\/sudo \/bin\/chown postgres \/dev\/watchdog (code=exited, status=0\/SUCCESS)\n  Process: 8468 ExecStartPre=\/usr\/bin\/sudo \/sbin\/modprobe softdog (code=exited, status=0\/SUCCESS)\n Main PID: 8482 (patroni)\n   CGroup: \/system.slice\/patroni.service\n           \u251c\u25008482 \/usr\/bin\/python2 \/u01\/app\/postgres\/local\/dmk\/bin\/patroni \/u01\/app\/postgres\/local\/dmk\/etc\/patroni.yml\n           \u251c\u25008500 \/u01\/app\/postgres\/product\/11\/db_5\/bin\/postgres -D \/u02\/pgdata\/11\/PG1\/ --config-file=\/u02\/pgdata\/11\/PG1\/postgresql.conf --listen_addresses=192.168.22.114 --max_worker_processes=8 --max_locks_per_transact...\n           \u251c\u25008502 postgres: PG1: logger\n           \u251c\u25008503 postgres: PG1: startup   waiting for 000000020000000000000006\n           \u251c\u25008504 postgres: PG1: checkpointer\n           \u251c\u25008505 postgres: PG1: background writer\n           \u251c\u25008506 postgres: PG1: stats collector\n           \u251c\u25008507 postgres: PG1: walreceiver\n           \u2514\u25008513 postgres: PG1: postgres postgres 192.168.22.114(48882) idle\n<\/pre>\n
To be sure, everything runs correctly, check the status of the Patroni cluster<\/p>\n
\npostgres@patroni4:\/home\/postgres\/ [PG1] patronictl list\n+---------+----------+----------------+--------+---------+----+-----------+\n| Cluster |  Member  |      Host      |  Role  |  State  | TL | Lag in MB |\n+---------+----------+----------------+--------+---------+----+-----------+\n|   PG1   | patroni1 | 192.168.22.111 |        | running |  2 |       0.0 |\n|   PG1   | patroni2 | 192.168.22.112 |        | running |  2 |       0.0 |\n|   PG1   | patroni3 | 192.168.22.113 | Leader | running |  2 |       0.0 |\n|   PG1   | patroni4 | 192.168.22.114 |        | running |  2 |       0.0 |\n+---------+----------+----------------+--------+---------+----+-----------+\n<\/pre>\n
 Conclusion <\/h3>\n
Using the playbooks had one failure. The entry for host in the patroni.yml is only checking localhost. When starting the fourth node, Patroni is not looking for all the other hosts, it is just looking for its own availability. This works fine in an initial cluster, but not when you want to extended one.
\nAnd: Always keep in mind, you need an uneven number of members for an etcd cluster, don’t add only a forth etcd node.<\/p>\n","protected":false},"excerpt":{"rendered":"
During the preparation of my presentation for the pgconf.eu I ran into one big issue. I had to stop my cluster to add a new node. That was not the way I wanted to archive this. I want a high availability solution, that can be scaled up without any outage. Due to a little hint […]<\/p>\n","protected":false},"author":28,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[229],"tags":[84,1543,586],"type_dbi":[],"class_list":["post-12975","post","type-post","status-publish","format-standard","hentry","category-database-administration-monitoring","tag-high-availability","tag-patroni","tag-postegresql"],"acf":[],"yoast_head":"\nHow to scale up a Patroni cluster - dbi Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to scale up a Patroni cluster\" \/>\n<meta property=\"og:description\" content=\"During the preparation of my presentation for the pgconf.eu I ran into one big issue. I had to stop my cluster to add a new node. That was not the way I wanted to archive this. I want a high availability solution, that can be scaled up without any outage. Due to a little hint […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2019-11-06T09:43:38+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-09-10T15:37:44+00:00\" \/>\n<meta name=\"author\" content=\"Open source Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Open source Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/\"},\"author\":{\"name\":\"Open source Team\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/59554f0d99383431eb6ed427e338952b\"},\"headline\":\"How to scale up a Patroni cluster\",\"datePublished\":\"2019-11-06T09:43:38+00:00\",\"dateModified\":\"2024-09-10T15:37:44+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/\"},\"wordCount\":407,\"commentCount\":0,\"keywords\":[\"High availability\",\"Patroni\",\"PostegreSQL\"],\"articleSection\":[\"Database Administration & Monitoring\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/\",\"name\":\"How to scale up a Patroni cluster - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"datePublished\":\"2019-11-06T09:43:38+00:00\",\"dateModified\":\"2024-09-10T15:37:44+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/59554f0d99383431eb6ed427e338952b\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to scale up a Patroni cluster\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/59554f0d99383431eb6ed427e338952b\",\"name\":\"Open source Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/eb4fb12e386e8c41fdef0733e8114594cf2653e4f55e9fa2161442b8eaf3f657?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/eb4fb12e386e8c41fdef0733e8114594cf2653e4f55e9fa2161442b8eaf3f657?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/eb4fb12e386e8c41fdef0733e8114594cf2653e4f55e9fa2161442b8eaf3f657?s=96&d=mm&r=g\",\"caption\":\"Open source Team\"},\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/open-source-team\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"How to scale up a Patroni cluster - dbi Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/","og_locale":"en_US","og_type":"article","og_title":"How to scale up a Patroni cluster","og_description":"During the preparation of my presentation for the pgconf.eu I ran into one big issue. I had to stop my cluster to add a new node. That was not the way I wanted to archive this. I want a high availability solution, that can be scaled up without any outage. Due to a little hint […]","og_url":"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/","og_site_name":"dbi Blog","article_published_time":"2019-11-06T09:43:38+00:00","article_modified_time":"2024-09-10T15:37:44+00:00","author":"Open source Team","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Open source Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/"},"author":{"name":"Open source Team","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/59554f0d99383431eb6ed427e338952b"},"headline":"How to scale up a Patroni cluster","datePublished":"2019-11-06T09:43:38+00:00","dateModified":"2024-09-10T15:37:44+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/"},"wordCount":407,"commentCount":0,"keywords":["High availability","Patroni","PostegreSQL"],"articleSection":["Database Administration & Monitoring"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/","url":"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/","name":"How to scale up a Patroni cluster - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2019-11-06T09:43:38+00:00","dateModified":"2024-09-10T15:37:44+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/59554f0d99383431eb6ed427e338952b"},"breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/how-to-scale-up-a-patroni-cluster\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"How to scale up a Patroni cluster"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/59554f0d99383431eb6ed427e338952b","name":"Open source Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/eb4fb12e386e8c41fdef0733e8114594cf2653e4f55e9fa2161442b8eaf3f657?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/eb4fb12e386e8c41fdef0733e8114594cf2653e4f55e9fa2161442b8eaf3f657?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/eb4fb12e386e8c41fdef0733e8114594cf2653e4f55e9fa2161442b8eaf3f657?s=96&d=mm&r=g","caption":"Open source Team"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/open-source-team\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/12975","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/28"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=12975"}],"version-history":[{"count":1,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/12975\/revisions"}],"predecessor-version":[{"id":34701,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/12975\/revisions\/34701"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=12975"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=12975"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=12975"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=12975"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}