{"id":11041,"date":"2018-03-21T16:32:14","date_gmt":"2018-03-21T15:32:14","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/"},"modified":"2018-03-21T16:32:14","modified_gmt":"2018-03-21T15:32:14","slug":"pg_basebackup-and-redirecting-progress-messages-to-a-file","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/","title":{"rendered":"pg_basebackup and redirecting progress messages to a file"},"content":{"rendered":"<p>Recently I came over that <a href=\"https:\/\/git.postgresql.org\/gitweb\/?p=postgresql.git;a=commit;h=143b54d21d37804707c27edebdbd4410891da133\" target=\"_blank\" rel=\"noopener\">commit<\/a> and wondered what that is about. The answer is quite simple but I didn&#8217;t know that this issue existed. Basically it is about how progress messages are written to screen and how they are written to a file. Lets have a look.<\/p>\n<p><!--more--><\/p>\n<p>When your run pg_basebackup with progress messages and in verbose mode the output looks like this:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1; highlight: [6]\">\npostgres@pgbox:\/home\/postgres\/ [PG10] pg_basebackup --pgdata=\/var\/tmp\/aa --verbose --progress \npg_basebackup: initiating base backup, waiting for checkpoint to complete\npg_basebackup: checkpoint completed\npg_basebackup: write-ahead log start point: 4\/30000028 on timeline 1\npg_basebackup: starting background WAL receiver\n593320\/593320 kB (100%), 1\/1 tablespace                                         \npg_basebackup: write-ahead log end point: 4\/30000130\npg_basebackup: waiting for background process to finish streaming ...\npg_basebackup: base backup completed\n<\/pre>\n<p>You&#8217;ll notice that the highlighted line always is overwritten on screen until we reach one hundred percent. Looking at that line when pg_basebackup is running will give you an estimate on how long it will take and you will see which file it is currently working on. When you do the same thing but kick it in the background like this:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\npostgres@pgbox:\/home\/postgres\/ [PG10] pg_basebackup --version\npg_basebackup (PostgreSQL) 10.0 \npostgres@pgbox:\/home\/postgres\/ [PG10] mkdir \/var\/tmp\/aa\npostgres@pgbox:\/home\/postgres\/ [PG10] nohup pg_basebackup --pgdata=\/var\/tmp\/aa --verbose --progress  &gt; \/tmp\/a.log 2&gt;&amp;1  &amp;\n<\/pre>\n<p>&#8230; you will have the same output in the log file:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\npostgres@pgbox:\/home\/postgres\/ [PG10] cat -f \/tmp\/a.log\nnohup: ignoring input\npg_basebackup: initiating base backup, waiting for checkpoint to complete\npg_basebackup: checkpoint completed\npg_basebackup: write-ahead log start point: 4\/28000028 on timeline 1\npg_basebackup: starting background WAL receiver\n593315\/593315 kB (100%), 1\/1 tablespace                                         \npg_basebackup: write-ahead log end point: 4\/28000130\npg_basebackup: waiting for background process to finish streaming ...\npg_basebackup: base backup completed\n<\/pre>\n<p>Somehow that was not considered very useful so the commit mentioned above changed that:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\npostgres@pgbox:\/home\/postgres\/ [PGDEV] pg_basebackup --version\npg_basebackup (PostgreSQL) 11devel\npostgres@pgbox:\/home\/postgres\/ [PGDEV] nohup pg_basebackup --pgdata=\/var\/tmp\/aa --verbose --progress  &gt; \/tmp\/a.log 2&gt;&amp;1  &amp;\npostgres@pgbox:\/home\/postgres\/ [PGDEV] cat \/tmp\/a.log\nnohup: ignoring input\npg_basebackup: initiating base backup, waiting for checkpoint to complete\npg_basebackup: checkpoint completed\npg_basebackup: write-ahead log start point: 0\/E000028 on timeline 1\npg_basebackup: starting background WAL receiver\npg_basebackup: created temporary replication slot \"pg_basebackup_29846\"\n     0\/184659 kB (0%), 0\/1 tablespace (\/var\/tmp\/aa\/backup_label           )\n  1705\/184659 kB (0%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/1\/1249            )\n  4697\/184659 kB (2%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/1\/2657            )\n  8395\/184659 kB (4%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/13276\/1255        )\n 20601\/184659 kB (11%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/13277\/2670        )\n 30614\/184659 kB (16%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/16395\/2607_vm     )\n 45367\/184659 kB (24%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/16395\/16424       )\n 54743\/184659 kB (29%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/16395\/16424       )\n 74327\/184659 kB (40%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/16395\/16424       )\n118807\/184659 kB (64%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/16395\/16424       )\n146647\/184659 kB (79%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/16395\/16424       )\n175197\/184659 kB (94%), 0\/1 tablespace (\/var\/tmp\/aa\/base\/16395\/16432       )\n184668\/184668 kB (100%), 0\/1 tablespace (\/var\/tmp\/aa\/global\/pg_control      )\n184668\/184668 kB (100%), 0\/1 tablespace (\/var\/tmp\/aa\/global\/pg_control      )\n184668\/184668 kB (100%), 1\/1 tablespace                                         \npg_basebackup: write-ahead log end point: 0\/E000168\npg_basebackup: waiting for background process to finish streaming ...\npg_basebackup: base backup completed\n<\/pre>\n<p>When you redirect the output into a file you can see much more steps in the log file compared to what we saw before (only a single line which is overwritten all the time). Seems to be a good change.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Recently I came over that commit and wondered what that is about. The answer is quite simple but I didn&#8217;t know that this issue existed. Basically it is about how progress messages are written to screen and how they are written to a file. Lets have a look.<\/p>\n","protected":false},"author":28,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[229],"tags":[77],"type_dbi":[],"class_list":["post-11041","post","type-post","status-publish","format-standard","hentry","category-database-administration-monitoring","tag-postgresql"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v28.8 (Yoast SEO v28.8) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>pg_basebackup and redirecting progress messages to a file - dbi Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"pg_basebackup and redirecting progress messages to a file\" \/>\n<meta property=\"og:description\" content=\"Recently I came over that commit and wondered what that is about. The answer is quite simple but I didn&#8217;t know that this issue existed. Basically it is about how progress messages are written to screen and how they are written to a file. Lets have a look.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2018-03-21T15:32:14+00:00\" \/>\n<meta name=\"author\" content=\"Open source Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Open source Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/pg_basebackup-and-redirecting-progress-messages-to-a-file\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/pg_basebackup-and-redirecting-progress-messages-to-a-file\\\/\"},\"author\":{\"name\":\"Open source Team\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/4bae5e5dd8c7d48edbd948772b49b202\"},\"headline\":\"pg_basebackup and redirecting progress messages to a file\",\"datePublished\":\"2018-03-21T15:32:14+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/pg_basebackup-and-redirecting-progress-messages-to-a-file\\\/\"},\"wordCount\":202,\"commentCount\":0,\"keywords\":[\"PostgreSQL\"],\"articleSection\":[\"Database Administration &amp; Monitoring\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/pg_basebackup-and-redirecting-progress-messages-to-a-file\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/pg_basebackup-and-redirecting-progress-messages-to-a-file\\\/\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/pg_basebackup-and-redirecting-progress-messages-to-a-file\\\/\",\"name\":\"pg_basebackup and redirecting progress messages to a file - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#website\"},\"datePublished\":\"2018-03-21T15:32:14+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/4bae5e5dd8c7d48edbd948772b49b202\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/pg_basebackup-and-redirecting-progress-messages-to-a-file\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/pg_basebackup-and-redirecting-progress-messages-to-a-file\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/pg_basebackup-and-redirecting-progress-messages-to-a-file\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"pg_basebackup and redirecting progress messages to a file\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/4bae5e5dd8c7d48edbd948772b49b202\",\"name\":\"Open source Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/eb4fb12e386e8c41fdef0733e8114594cf2653e4f55e9fa2161442b8eaf3f657?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/eb4fb12e386e8c41fdef0733e8114594cf2653e4f55e9fa2161442b8eaf3f657?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/eb4fb12e386e8c41fdef0733e8114594cf2653e4f55e9fa2161442b8eaf3f657?s=96&d=mm&r=g\",\"caption\":\"Open source Team\"},\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/author\\\/open-source-team\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"pg_basebackup and redirecting progress messages to a file - dbi Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/","og_locale":"en_US","og_type":"article","og_title":"pg_basebackup and redirecting progress messages to a file","og_description":"Recently I came over that commit and wondered what that is about. The answer is quite simple but I didn&#8217;t know that this issue existed. Basically it is about how progress messages are written to screen and how they are written to a file. Lets have a look.","og_url":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/","og_site_name":"dbi Blog","article_published_time":"2018-03-21T15:32:14+00:00","author":"Open source Team","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Open source Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/"},"author":{"name":"Open source Team","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/4bae5e5dd8c7d48edbd948772b49b202"},"headline":"pg_basebackup and redirecting progress messages to a file","datePublished":"2018-03-21T15:32:14+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/"},"wordCount":202,"commentCount":0,"keywords":["PostgreSQL"],"articleSection":["Database Administration &amp; Monitoring"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/","url":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/","name":"pg_basebackup and redirecting progress messages to a file - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2018-03-21T15:32:14+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/4bae5e5dd8c7d48edbd948772b49b202"},"breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/pg_basebackup-and-redirecting-progress-messages-to-a-file\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"pg_basebackup and redirecting progress messages to a file"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/4bae5e5dd8c7d48edbd948772b49b202","name":"Open source Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/eb4fb12e386e8c41fdef0733e8114594cf2653e4f55e9fa2161442b8eaf3f657?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/eb4fb12e386e8c41fdef0733e8114594cf2653e4f55e9fa2161442b8eaf3f657?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/eb4fb12e386e8c41fdef0733e8114594cf2653e4f55e9fa2161442b8eaf3f657?s=96&d=mm&r=g","caption":"Open source Team"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/open-source-team\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/11041","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/28"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=11041"}],"version-history":[{"count":0,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/11041\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=11041"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=11041"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=11041"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=11041"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}