{"id":11041,"date":"2018-03-21T16:32:14","date_gmt":"2018-03-21T15:32:14","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/"},"modified":"2018-03-21T16:32:14","modified_gmt":"2018-03-21T15:32:14","slug":"pg_basebackup-and-redirecting-progress-messages-to-a-file","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/","title":{"rendered":"pg_basebackup and redirecting progress messages to a file"},"content":{"rendered":"<p>Recently I came over that <a href=\"https:\/\/git.postgresql.org\/gitweb\/?p=postgresql.git;a=commit;h=143b54d21d37804707c27edebdbd4410891da133\" target=\"_blank\" rel=\"noopener\">commit<\/a> and wondered what that is about. The answer is quite simple but I didn&#8217;t know that this issue existed. Basically it is about how progress messages are written to screen and how they are written to a file. Lets have a look.<\/p>\n<p><!--more--><\/p>\n<p>When your run pg_basebackup with progress messages and in verbose mode the output looks like this:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1; highlight: [6]\">\npostgres@pgbox:\/home\/postgres\/ [PG10] pg_basebackup --pgdata=\/var\/tmp\/aa --verbose --progress \npg_basebackup: initiating base backup, waiting for checkpoint to complete\npg_basebackup: checkpoint completed\npg_basebackup: write-ahead log start point: 4\/30000028 on timeline 1\npg_basebackup: starting background WAL receiver\n593320\/593320 kB (100%), 1\/1 tablespace                                         \npg_basebackup: write-ahead log end point: 4\/30000130\npg_basebackup: waiting for background process to finish streaming ...\npg_basebackup: base backup completed\n<\/pre>\n<p>You&#8217;ll notice that the highlighted line always is overwritten on screen until we reach one hundred percent. Looking at that line when pg_basebackup is running will give you an estimate on how long it will take and you will see which file it is currently working on. When you do the same thing but kick it in the background like this:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\npostgres@pgbox:\/home\/postgres\/ [PG10] pg_basebackup --version\npg_basebackup (PostgreSQL) 10.0 \npostgres@pgbox:\/home\/postgres\/ [PG10] mkdir \/var\/tmp\/aa\npostgres@pgbox:\/home\/postgres\/ [PG10] nohup pg_basebackup --pgdata=\/var\/tmp\/aa --verbose --progress  &gt; \/tmp\/a.log 2&gt;&amp;1  &amp;\n<\/pre>\n<p>&#8230; you will have the same output in the log file:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\npostgres@pgbox:\/home\/postgres\/ [PG10] cat -f \/tmp\/a.log\nnohup: ignoring input\npg_basebackup: initiating base backup, waiting for checkpoint to complete\npg_basebackup: checkpoint completed\npg_basebackup: write-ahead log start point: 4\/28000028 on timeline 1\npg_basebackup: starting background WAL receiver\n593315\/593315 kB (100%), 1\/1 tablespace                                         \npg_basebackup: write-ahead log end point: 4\/28000130\npg_basebackup: waiting for background process to finish streaming ...\npg_basebackup: base backup completed\n<\/pre>\n<p>Somehow that was not considered very useful so the commit mentioned above changed that:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\npostgres@pgbox:\/home\/postgres\/ [PGDEV] pg_basebackup --version\npg_basebackup (PostgreSQL) 11devel\npostgres@pgbox:\/home\/postgres\/ [PGDEV] nohup pg_basebackup --pgdata=\/var\/tmp\/aa --verbose --progress  &gt; \/tmp\/a.log 2&gt;&amp;1  &amp;\npostgres@pgbox:\/home\/postgres\/ [PGDEV] cat \/tmp\/a.log\nnohup: ignoring input\npg_basebackup: initiating base backup, waiting for checkpoint to complete\npg_basebackup: checkpoint completed\npg_basebackup: write-ahead log start point: 0\/E000028 on timeline 1\npg_basebackup: starting background WAL receiver\npg_basebackup: created temporary replication slot \"pg_basebackup_29846\"\n     0\/184659 kB (0%), 0\/1 tablespace (\/var\/tmp\/aa\/backup_label           )\n  1705\/184659 kB (0%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/1\/1249            )\n  4697\/184659 kB (2%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/1\/2657            )\n  8395\/184659 kB (4%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/13276\/1255        )\n 20601\/184659 kB (11%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/13277\/2670        )\n 30614\/184659 kB (16%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/16395\/2607_vm     )\n 45367\/184659 kB (24%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/16395\/16424       )\n 54743\/184659 kB (29%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/16395\/16424       )\n 74327\/184659 kB (40%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/16395\/16424       )\n118807\/184659 kB (64%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/16395\/16424       )\n146647\/184659 kB (79%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/16395\/16424       )\n175197\/184659 kB (94%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/16395\/16432       )\n184668\/184668 kB (100%), 0\/1 tablespace (\/var\/tmp\/aa\/global\/pg_control      )\n184668\/184668 kB (100%), 0\/1 tablespace (\/var\/tmp\/aa\/global\/pg_control      )\n184668\/184668 kB (100%), 1\/1 tablespace                                         \npg_basebackup: write-ahead log end point: 0\/E000168\npg_basebackup: waiting for background process to finish streaming ...\npg_basebackup: base backup completed\n<\/pre>\n<p>When you redirect the output into a file you can see much more steps in the log file compared to what we saw before (only a single line which is overwritten all the time). Seems to be a good change.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Recently I came over that commit and wondered what that is about. The answer is quite simple but I didn&#8217;t know that this issue existed. Basically it is about how progress messages are written to screen and how they are written to a file. Lets have a look.<\/p>\n","protected":false},"author":29,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[229],"tags":[77],"type_dbi":[],"class_list":["post-11041","post","type-post","status-publish","format-standard","hentry","category-database-administration-monitoring","tag-postgresql"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.2 (Yoast SEO v27.2) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>pg_basebackup and redirecting progress messages to a file - dbi Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"pg_basebackup and redirecting progress messages to a file\" \/>\n<meta property=\"og:description\" content=\"Recently I came over that commit and wondered what that is about. The answer is quite simple but I didn&#8217;t know that this issue existed. Basically it is about how progress messages are written to screen and how they are written to a file. Lets have a look.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2018-03-21T15:32:14+00:00\" \/>\n<meta name=\"author\" content=\"Daniel Westermann\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@westermanndanie\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Daniel Westermann\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/\"},\"author\":{\"name\":\"Daniel Westermann\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66\"},\"headline\":\"pg_basebackup and redirecting progress messages to a file\",\"datePublished\":\"2018-03-21T15:32:14+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/\"},\"wordCount\":202,\"commentCount\":0,\"keywords\":[\"PostgreSQL\"],\"articleSection\":[\"Database Administration &amp; Monitoring\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/\",\"name\":\"pg_basebackup and redirecting progress messages to a file - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"datePublished\":\"2018-03-21T15:32:14+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"pg_basebackup and redirecting progress messages to a file\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66\",\"name\":\"Daniel Westermann\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g\",\"caption\":\"Daniel Westermann\"},\"description\":\"Daniel Westermann is Principal Consultant and Technology Leader Open Infrastructure at dbi services. He has more than 15 years of experience in management, engineering and optimization of databases and infrastructures, especially on Oracle and PostgreSQL. Since the beginning of his career, he has specialized in Oracle Technologies and is Oracle Certified Professional 12c and Oracle Certified Expert RAC\/GridInfra. Over time, Daniel has become increasingly interested in open source technologies, becoming \u201cTechnology Leader Open Infrastructure\u201d and PostgreSQL expert. \u00a0Based on community or EnterpriseDB tools, he develops and installs complex high available solutions with PostgreSQL. He is also a certified PostgreSQL Plus 9.0 Professional and a Postgres Advanced Server 9.4 Professional. He is a regular speaker at PostgreSQL conferences in Switzerland and Europe. Today Daniel is also supporting our customers on AWS services such as AWS RDS, database migrations into the cloud, EC2 and automated infrastructure management with AWS SSM (System Manager). He is a certified AWS Solutions Architect Professional. Prior to dbi services, Daniel was Management System Engineer at LC SYSTEMS-Engineering AG in Basel. Before that, he worked as Oracle Developper &amp;\u00a0Project Manager at Delta Energy Solutions AG in Basel (today Powel AG). Daniel holds a diploma in Business Informatics (DHBW, Germany). His branch-related experience mainly covers the pharma industry, the financial sector, energy, lottery and telecommunications.\",\"sameAs\":[\"https:\/\/x.com\/westermanndanie\"],\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/daniel-westermann\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"pg_basebackup and redirecting progress messages to a file - dbi Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/","og_locale":"en_US","og_type":"article","og_title":"pg_basebackup and redirecting progress messages to a file","og_description":"Recently I came over that commit and wondered what that is about. The answer is quite simple but I didn&#8217;t know that this issue existed. Basically it is about how progress messages are written to screen and how they are written to a file. Lets have a look.","og_url":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/","og_site_name":"dbi Blog","article_published_time":"2018-03-21T15:32:14+00:00","author":"Daniel Westermann","twitter_card":"summary_large_image","twitter_creator":"@westermanndanie","twitter_misc":{"Written by":"Daniel Westermann","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/"},"author":{"name":"Daniel Westermann","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66"},"headline":"pg_basebackup and redirecting progress messages to a file","datePublished":"2018-03-21T15:32:14+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/"},"wordCount":202,"commentCount":0,"keywords":["PostgreSQL"],"articleSection":["Database Administration &amp; Monitoring"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/","url":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/","name":"pg_basebackup and redirecting progress messages to a file - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2018-03-21T15:32:14+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66"},"breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"pg_basebackup and redirecting progress messages to a file"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66","name":"Daniel Westermann","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g","caption":"Daniel Westermann"},"description":"Daniel Westermann is Principal Consultant and Technology Leader Open Infrastructure at dbi services. He has more than 15 years of experience in management, engineering and optimization of databases and infrastructures, especially on Oracle and PostgreSQL. Since the beginning of his career, he has specialized in Oracle Technologies and is Oracle Certified Professional 12c and Oracle Certified Expert RAC\/GridInfra. Over time, Daniel has become increasingly interested in open source technologies, becoming \u201cTechnology Leader Open Infrastructure\u201d and PostgreSQL expert. \u00a0Based on community or EnterpriseDB tools, he develops and installs complex high available solutions with PostgreSQL. He is also a certified PostgreSQL Plus 9.0 Professional and a Postgres Advanced Server 9.4 Professional. He is a regular speaker at PostgreSQL conferences in Switzerland and Europe. Today Daniel is also supporting our customers on AWS services such as AWS RDS, database migrations into the cloud, EC2 and automated infrastructure management with AWS SSM (System Manager). He is a certified AWS Solutions Architect Professional. Prior to dbi services, Daniel was Management System Engineer at LC SYSTEMS-Engineering AG in Basel. Before that, he worked as Oracle Developper &amp;\u00a0Project Manager at Delta Energy Solutions AG in Basel (today Powel AG). Daniel holds a diploma in Business Informatics (DHBW, Germany). His branch-related experience mainly covers the pharma industry, the financial sector, energy, lottery and telecommunications.","sameAs":["https:\/\/x.com\/westermanndanie"],"url":"https:\/\/www.dbi-services.com\/blog\/author\/daniel-westermann\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/11041","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/29"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=11041"}],"version-history":[{"count":0,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/11041\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=11041"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=11041"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=11041"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=11041"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}