{"id":35182,"date":"2024-10-11T07:32:48","date_gmt":"2024-10-11T05:32:48","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/?p=35182"},"modified":"2024-10-11T07:32:51","modified_gmt":"2024-10-11T05:32:51","slug":"postgresql-18-reject_limit-for-copy","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/","title":{"rendered":"PostgreSQL 18: REJECT_LIMIT for COPY"},"content":{"rendered":"\n<p>With the just released <a href=\"https:\/\/www.postgresql.org\/about\/news\/postgresql-17-released-2936\/\" target=\"_blank\" rel=\"noreferrer noopener\">version of 17 of PostgreSQL<\/a>, <a href=\"https:\/\/www.postgresql.org\/docs\/17\/sql-copy.html\" target=\"_blank\" rel=\"noreferrer noopener\">copy<\/a> got the &#8220;ON_ERROR&#8221; switch.  This is very useful if you load something which contains rows that cannot be inserted into the target table. Before that, copy would just stop and error out. The only option was to fix the data and then copy again. Using on &#8220;ON_ERROR&#8221;, copy can be told to continue loading and ignore those rows with bad data.<\/p>\n\n\n\n<p>The default behavior is still to stop as soon as there is a row\/line which cannot be loaded:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: sql; highlight: [1,3,8]; title: ; notranslate\" title=\"\">\npostgres=# create table t ( a int, b int, c text );\nCREATE TABLE\npostgres=# \\! cat load.txt\n1,1,&#039;aaa&#039;\n2,2,&#039;bbb&#039;\n3,3,&#039;ccc&#039;\n4,four,&#039;ddd&#039;\n5,5,&#039;eee&#039;\n6,6,&#039;fff&#039;\n7,7,&#039;ggg&#039;\n8,eight,&#039;hhh&#039;\n9,9,&#039;iii&#039;\npostgres=#  copy t from &#039;\/home\/postgres\/load.txt&#039; with ( delimiter &#039;,&#039;);\nERROR:  invalid input syntax for type integer: &quot;four&quot;\nCONTEXT:  COPY t, line 4, column b: &quot;four&quot;\n<\/pre><\/div>\n\n\n<p>Starting with PostgreSQL 17, we can tell copy to ignore those rows and continue to load the data:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: sql; highlight: [0,1,3]; title: ; notranslate\" title=\"\">\npostgres=#  copy t from &#039;\/home\/postgres\/load.txt&#039; with ( delimiter &#039;,&#039;, on_error ignore);\nCOPY 7\npostgres=# select * from t;\n a | b |   c   \n---+---+-------\n 1 | 1 | &#039;aaa&#039;\n 2 | 2 | &#039;bbb&#039;\n 3 | 3 | &#039;ccc&#039;\n 5 | 5 | &#039;eee&#039;\n 6 | 6 | &#039;fff&#039;\n 7 | 7 | &#039;ggg&#039;\n 9 | 9 | &#039;iii&#039;\n(7 rows)\n<\/pre><\/div>\n\n\n<p>What we cannot do up to PostgreSQL 17 is to set a limit for bad data. This will be possible starting with PostgreSQL 18, as copy got a new switch: &#8220;REJECT_LIMIT&#8221;.<\/p>\n\n\n\n<p>Using this, we can tell copy to stop as soon as we reach n-rows which cannot be loaded:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: sql; highlight: [1,3,5,7,9,11]; title: ; notranslate\" title=\"\">\npostgres=# truncate t;\nTRUNCATE TABLE\npostgres=#  copy t from &#039;\/home\/postgres\/load.txt&#039; with ( delimiter &#039;,&#039;, on_error ignore, reject_limit 1);\nERROR:  skipped more than REJECT_LIMIT (1) rows due to data type incompatibility\nCONTEXT:  COPY t, line 8, column b: &quot;eight&quot;\npostgres=#  copy t from &#039;\/home\/postgres\/load.txt&#039; with ( delimiter &#039;,&#039;, on_error ignore, reject_limit 2);\nERROR:  skipped more than REJECT_LIMIT (2) rows due to data type incompatibility\nCONTEXT:  COPY t, line 10, column a: &quot;&quot;\npostgres=#  copy t from &#039;\/home\/postgres\/load.txt&#039; with ( delimiter &#039;,&#039;, on_error ignore, reject_limit 3);\nCOPY 7\npostgres=# select * from t;\n a | b |   c   \n---+---+-------\n 1 | 1 | &#039;aaa&#039;\n 2 | 2 | &#039;bbb&#039;\n 3 | 3 | &#039;ccc&#039;\n 5 | 5 | &#039;eee&#039;\n 6 | 6 | &#039;fff&#039;\n 7 | 7 | &#039;ggg&#039;\n 9 | 9 | &#039;iii&#039;\n(7 rows)\n<\/pre><\/div>\n\n\n<p>Nice, this gives more control about how to handle data which does not fit into into the target table.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>With the just released version of 17 of PostgreSQL, copy got the &#8220;ON_ERROR&#8221; switch. This is very useful if you load something which contains rows that cannot be inserted into the target table. Before that, copy would just stop and error out. The only option was to fix the data and then copy again. Using [&hellip;]<\/p>\n","protected":false},"author":29,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[229,198],"tags":[77],"type_dbi":[],"class_list":["post-35182","post","type-post","status-publish","format-standard","hentry","category-database-administration-monitoring","category-database-management","tag-postgresql"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.2 (Yoast SEO v27.2) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>PostgreSQL 18: REJECT_LIMIT for COPY - dbi Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"PostgreSQL 18: REJECT_LIMIT for COPY\" \/>\n<meta property=\"og:description\" content=\"With the just released version of 17 of PostgreSQL, copy got the &#8220;ON_ERROR&#8221; switch. This is very useful if you load something which contains rows that cannot be inserted into the target table. Before that, copy would just stop and error out. The only option was to fix the data and then copy again. Using [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2024-10-11T05:32:48+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-10-11T05:32:51+00:00\" \/>\n<meta name=\"author\" content=\"Daniel Westermann\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@westermanndanie\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Daniel Westermann\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/\"},\"author\":{\"name\":\"Daniel Westermann\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66\"},\"headline\":\"PostgreSQL 18: REJECT_LIMIT for COPY\",\"datePublished\":\"2024-10-11T05:32:48+00:00\",\"dateModified\":\"2024-10-11T05:32:51+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/\"},\"wordCount\":180,\"commentCount\":0,\"keywords\":[\"PostgreSQL\"],\"articleSection\":[\"Database Administration &amp; Monitoring\",\"Database management\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/\",\"name\":\"PostgreSQL 18: REJECT_LIMIT for COPY - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"datePublished\":\"2024-10-11T05:32:48+00:00\",\"dateModified\":\"2024-10-11T05:32:51+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"PostgreSQL 18: REJECT_LIMIT for COPY\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66\",\"name\":\"Daniel Westermann\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g\",\"caption\":\"Daniel Westermann\"},\"description\":\"Daniel Westermann is Principal Consultant and Technology Leader Open Infrastructure at dbi services. He has more than 15 years of experience in management, engineering and optimization of databases and infrastructures, especially on Oracle and PostgreSQL. Since the beginning of his career, he has specialized in Oracle Technologies and is Oracle Certified Professional 12c and Oracle Certified Expert RAC\/GridInfra. Over time, Daniel has become increasingly interested in open source technologies, becoming \u201cTechnology Leader Open Infrastructure\u201d and PostgreSQL expert. \u00a0Based on community or EnterpriseDB tools, he develops and installs complex high available solutions with PostgreSQL. He is also a certified PostgreSQL Plus 9.0 Professional and a Postgres Advanced Server 9.4 Professional. He is a regular speaker at PostgreSQL conferences in Switzerland and Europe. Today Daniel is also supporting our customers on AWS services such as AWS RDS, database migrations into the cloud, EC2 and automated infrastructure management with AWS SSM (System Manager). He is a certified AWS Solutions Architect Professional. Prior to dbi services, Daniel was Management System Engineer at LC SYSTEMS-Engineering AG in Basel. Before that, he worked as Oracle Developper &amp;\u00a0Project Manager at Delta Energy Solutions AG in Basel (today Powel AG). Daniel holds a diploma in Business Informatics (DHBW, Germany). His branch-related experience mainly covers the pharma industry, the financial sector, energy, lottery and telecommunications.\",\"sameAs\":[\"https:\/\/x.com\/westermanndanie\"],\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/daniel-westermann\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"PostgreSQL 18: REJECT_LIMIT for COPY - dbi Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/","og_locale":"en_US","og_type":"article","og_title":"PostgreSQL 18: REJECT_LIMIT for COPY","og_description":"With the just released version of 17 of PostgreSQL, copy got the &#8220;ON_ERROR&#8221; switch. This is very useful if you load something which contains rows that cannot be inserted into the target table. Before that, copy would just stop and error out. The only option was to fix the data and then copy again. Using [&hellip;]","og_url":"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/","og_site_name":"dbi Blog","article_published_time":"2024-10-11T05:32:48+00:00","article_modified_time":"2024-10-11T05:32:51+00:00","author":"Daniel Westermann","twitter_card":"summary_large_image","twitter_creator":"@westermanndanie","twitter_misc":{"Written by":"Daniel Westermann","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/"},"author":{"name":"Daniel Westermann","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66"},"headline":"PostgreSQL 18: REJECT_LIMIT for COPY","datePublished":"2024-10-11T05:32:48+00:00","dateModified":"2024-10-11T05:32:51+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/"},"wordCount":180,"commentCount":0,"keywords":["PostgreSQL"],"articleSection":["Database Administration &amp; Monitoring","Database management"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/","url":"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/","name":"PostgreSQL 18: REJECT_LIMIT for COPY - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2024-10-11T05:32:48+00:00","dateModified":"2024-10-11T05:32:51+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66"},"breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/postgresql-18-reject_limit-for-copy\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"PostgreSQL 18: REJECT_LIMIT for COPY"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66","name":"Daniel Westermann","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g","caption":"Daniel Westermann"},"description":"Daniel Westermann is Principal Consultant and Technology Leader Open Infrastructure at dbi services. He has more than 15 years of experience in management, engineering and optimization of databases and infrastructures, especially on Oracle and PostgreSQL. Since the beginning of his career, he has specialized in Oracle Technologies and is Oracle Certified Professional 12c and Oracle Certified Expert RAC\/GridInfra. Over time, Daniel has become increasingly interested in open source technologies, becoming \u201cTechnology Leader Open Infrastructure\u201d and PostgreSQL expert. \u00a0Based on community or EnterpriseDB tools, he develops and installs complex high available solutions with PostgreSQL. He is also a certified PostgreSQL Plus 9.0 Professional and a Postgres Advanced Server 9.4 Professional. He is a regular speaker at PostgreSQL conferences in Switzerland and Europe. Today Daniel is also supporting our customers on AWS services such as AWS RDS, database migrations into the cloud, EC2 and automated infrastructure management with AWS SSM (System Manager). He is a certified AWS Solutions Architect Professional. Prior to dbi services, Daniel was Management System Engineer at LC SYSTEMS-Engineering AG in Basel. Before that, he worked as Oracle Developper &amp;\u00a0Project Manager at Delta Energy Solutions AG in Basel (today Powel AG). Daniel holds a diploma in Business Informatics (DHBW, Germany). His branch-related experience mainly covers the pharma industry, the financial sector, energy, lottery and telecommunications.","sameAs":["https:\/\/x.com\/westermanndanie"],"url":"https:\/\/www.dbi-services.com\/blog\/author\/daniel-westermann\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/35182","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/29"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=35182"}],"version-history":[{"count":5,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/35182\/revisions"}],"predecessor-version":[{"id":35187,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/35182\/revisions\/35187"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=35182"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=35182"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=35182"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=35182"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}