{"id":19092,"date":"2022-09-22T19:04:21","date_gmt":"2022-09-22T17:04:21","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/?p=19092"},"modified":"2022-09-22T19:04:23","modified_gmt":"2022-09-22T17:04:23","slug":"toasting-in-postgresql-lets-see-it-in-action","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/","title":{"rendered":"TOASTing in PostgreSQL, let’s see it in action"},"content":{"rendered":"

The last two posts introduced TOASTing strategies<\/a> and how TOAST tables<\/a> look like. Have a look at those posts if you are not yet familiar with TOAST in PostgreSQL. In this post we’ll look a bit under the hood to understand how TOASTing really works.
\n
\nWe’ll use the same table as in the last post, which is still there but empty right now:<\/p>\n

postgres=# \\d t\n                 Table \"public.t\"\n Column |  Type   | Collation | Nullable | Default \n--------+---------+-----------+----------+---------\n a      | integer |           |          | \n b      | text    |           |          | \n\npostgres=# \\d :reltoastrelid\nTOAST table \"pg_toast.pg_toast_16418\"\n   Column   |  Type   \n------------+---------\n chunk_id   | oid\n chunk_seq  | integer\n chunk_data | bytea\nOwning table: \"public.t\"\nIndexes:\n    \"pg_toast_16418_index\" PRIMARY KEY, btree (chunk_id, chunk_seq)\n\npostgres=# select count(*) from t;\n count \n-------\n     0\n(1 row)\n<\/pre>\n
What happens if insert a row like this:<\/p>\n
postgres=# insert into t values (1, repeat('x',10000));\nINSERT 0 1\n<\/pre>\n
The second column now contains a string with a length of 10000 characters. Did this trigger the TOASTer?<\/p>\n
postgres=# select count(*) from :reltoastrelid;\n count \n-------\n     0\n(1 row)\n<\/pre>\n
It did not, and the reason is easy to explain: This string can easily be compressed and there is no need to move it out of line. This leads us to another question: What compression algorithm was used to compress the data? The default is this:<\/p>\n
postgres=# show default_toast_compression;\n default_toast_compression \n---------------------------\n pglz\n(1 row)\n<\/pre>\n
Starting with PostgreSQL 14 you have an additional option:<\/p>\n
postgres=# set default_toast_compression = lz4;\nSET\n<\/pre>\n
LZ4<\/a> “is focused on compression and decompression speed” and usually gives you better results than the buid-in PGLZ algorithm. Support for LZ4 needs to be compiled in, otherwise it is not available. The packages provided by the PostgreSQL community have it enabled by default, if you compile from source you need to enable it:<\/p>\n
postgres@debian11pg:\/home\/postgres\/postgresql\/ [pg16] .\/configure --help | grep lz4\n  --with-lz4              build with LZ4 support\n<\/pre>\n
You also have the flexibility to change the compression algorithm on a column basis:<\/p>\n
postgres=# alter table t alter column b set compression lz4;\nALTER TABLE\n<\/pre>\n
As soon as you change the default compression algorithm for a column, this is recorded in pg_attribute<\/a> (it is empty if the default compression is used):<\/p>\n
postgres=# select attcompression from pg_attribute where attname = 'b';\n attcompression \n----------------\n l\n(1 row)\n<\/pre>\n
The performance benefit of LZ4 compared to PGLZ is quite huge, as you can see in the little test below:<\/p>\n
postgres=# \\timing\nTiming is on.\npostgres=# truncate t;\nTRUNCATE TABLE\nTime: 18.493 ms\npostgres=# insert into t select i, repeat('x',10000) from generate_series(1,1000000) i;\nINSERT 0 1000000\nTime: 3431.030 ms (00:03.431)\npostgres=# select pg_size_pretty(pg_total_relation_size('t'));\n pg_size_pretty \n----------------\n 89 MB\n(1 row)\nTime: 1.361 ms\npostgres=# truncate t;\nTRUNCATE TABLE\nTime: 45.045 ms\npostgres=# alter table t alter column b set compression pglz;\nALTER TABLE\nTime: 13.442 ms\npostgres=# insert into t select i, repeat('x',10000) from generate_series(1,1000000) i;\nINSERT 0 1000000\nTime: 51065.485 ms (00:51.065)\npostgres=# select pg_size_pretty(pg_total_relation_size('t'));\n pg_size_pretty \n----------------\n 160 MB\n(1 row)\n<\/pre>\n
How can we force the TOASTer to kick in? One way of doing that is by generating a string that is not easily compressible like this:<\/p>\n
postgres=# truncate t;\nTRUNCATE TABLE\npostgres=# with dummy_string as\n( select string_agg (md5(random()::text),'') as dummy \n    from generate_series(1,5000) )\n  insert into t \n  select 1\n       , dummy_string.dummy \n    from dummy_string;\nINSERT 0 1\n<\/pre>\n
This will give us 81 chunks in the corresponding TOAST table:<\/p>\n
postgres=# select count(*) from :reltoastrelid;\n count \n-------\n    81\n(1 row)\n<\/pre>\n
… and the data looks like this:<\/p>\n
postgres=# select * from :reltoastrelid limit 3;\n chunk_id | chunk_seq |                                                                                                      >\n----------+-----------+------------------------------------------------------------------------------------------------------>\n    16455 |         0 | \\x313839623233643865653830316434323863336230333463633731313435333766396133393130356262383739343765373>\n    16455 |         1 | \\x313461633239646532633936613433306664636664303337346332343737626563336266373663346435386563633166663>\n    16455 |         2 | \\x633262373562616563393536636537383537323434633565363639396337646164653138623430366635633265383135643>\n(3 rows)\n<\/pre>\n
We managed to trigger the TOASTer. What still needs to be answered is: At what size of a row is the TOASTer triggered. The answer is in “\/src\/include\/access\/heaptoast.h”: If a new tuple is larger than 2000 bytes, then the TOASTer kicks in and does its job.<\/p>\n
Some final words: You never should use “select *” against tables anyway, but especially against tables which contain data which is moved out-of-line. If you do that, you will always pull out the TOASTed data, and this might take some time. Only retrieve it, if you really need it.<\/p>\n","protected":false},"excerpt":{"rendered":"
The last two posts introduced TOASTing strategies and how TOAST tables look like. Have a look at those posts if you are not yet familiar with TOAST in PostgreSQL. In this post we’ll look a bit under the hood to understand how TOASTing really works.<\/p>\n","protected":false},"author":29,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[229],"tags":[2602],"type_dbi":[],"class_list":["post-19092","post","type-post","status-publish","format-standard","hentry","category-database-administration-monitoring","tag-postgresql-2"],"acf":[],"yoast_head":"\nTOASTing in PostgreSQL, let's see it in action - dbi Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"TOASTing in PostgreSQL, let's see it in action\" \/>\n<meta property=\"og:description\" content=\"The last two posts introduced TOASTing strategies and how TOAST tables look like. Have a look at those posts if you are not yet familiar with TOAST in PostgreSQL. In this post we’ll look a bit under the hood to understand how TOASTing really works.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2022-09-22T17:04:21+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-09-22T17:04:23+00:00\" \/>\n<meta name=\"author\" content=\"Daniel Westermann\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@westermanndanie\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Daniel Westermann\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/\"},\"author\":{\"name\":\"Daniel Westermann\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66\"},\"headline\":\"TOASTing in PostgreSQL, let’s see it in action\",\"datePublished\":\"2022-09-22T17:04:21+00:00\",\"dateModified\":\"2022-09-22T17:04:23+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/\"},\"wordCount\":407,\"commentCount\":0,\"keywords\":[\"postgresql\"],\"articleSection\":[\"Database Administration & Monitoring\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/\",\"name\":\"TOASTing in PostgreSQL, let's see it in action - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"datePublished\":\"2022-09-22T17:04:21+00:00\",\"dateModified\":\"2022-09-22T17:04:23+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"TOASTing in PostgreSQL, let’s see it in action\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66\",\"name\":\"Daniel Westermann\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g\",\"caption\":\"Daniel Westermann\"},\"description\":\"Daniel Westermann is Principal Consultant and Technology Leader Open Infrastructure at dbi services. He has more than 15 years of experience in management, engineering and optimization of databases and infrastructures, especially on Oracle and PostgreSQL. Since the beginning of his career, he has specialized in Oracle Technologies and is Oracle Certified Professional 12c and Oracle Certified Expert RAC\/GridInfra. Over time, Daniel has become increasingly interested in open source technologies, becoming \u201cTechnology Leader Open Infrastructure\u201d and PostgreSQL expert. \u00a0Based on community or EnterpriseDB tools, he develops and installs complex high available solutions with PostgreSQL. He is also a certified PostgreSQL Plus 9.0 Professional and a Postgres Advanced Server 9.4 Professional. He is a regular speaker at PostgreSQL conferences in Switzerland and Europe. Today Daniel is also supporting our customers on AWS services such as AWS RDS, database migrations into the cloud, EC2 and automated infrastructure management with AWS SSM (System Manager). He is a certified AWS Solutions Architect Professional. Prior to dbi services, Daniel was Management System Engineer at LC SYSTEMS-Engineering AG in Basel. Before that, he worked as Oracle Developper &\u00a0Project Manager at Delta Energy Solutions AG in Basel (today Powel AG). Daniel holds a diploma in Business Informatics (DHBW, Germany). His branch-related experience mainly covers the pharma industry, the financial sector, energy, lottery and telecommunications.\",\"sameAs\":[\"https:\/\/x.com\/westermanndanie\"],\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/daniel-westermann\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"TOASTing in PostgreSQL, let's see it in action - dbi Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/","og_locale":"en_US","og_type":"article","og_title":"TOASTing in PostgreSQL, let's see it in action","og_description":"The last two posts introduced TOASTing strategies and how TOAST tables look like. Have a look at those posts if you are not yet familiar with TOAST in PostgreSQL. In this post we’ll look a bit under the hood to understand how TOASTing really works.","og_url":"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/","og_site_name":"dbi Blog","article_published_time":"2022-09-22T17:04:21+00:00","article_modified_time":"2022-09-22T17:04:23+00:00","author":"Daniel Westermann","twitter_card":"summary_large_image","twitter_creator":"@westermanndanie","twitter_misc":{"Written by":"Daniel Westermann","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/"},"author":{"name":"Daniel Westermann","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66"},"headline":"TOASTing in PostgreSQL, let’s see it in action","datePublished":"2022-09-22T17:04:21+00:00","dateModified":"2022-09-22T17:04:23+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/"},"wordCount":407,"commentCount":0,"keywords":["postgresql"],"articleSection":["Database Administration & Monitoring"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/","url":"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/","name":"TOASTing in PostgreSQL, let's see it in action - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2022-09-22T17:04:21+00:00","dateModified":"2022-09-22T17:04:23+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66"},"breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/toasting-in-postgresql-lets-see-it-in-action\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"TOASTing in PostgreSQL, let’s see it in action"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66","name":"Daniel Westermann","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g","caption":"Daniel Westermann"},"description":"Daniel Westermann is Principal Consultant and Technology Leader Open Infrastructure at dbi services. He has more than 15 years of experience in management, engineering and optimization of databases and infrastructures, especially on Oracle and PostgreSQL. Since the beginning of his career, he has specialized in Oracle Technologies and is Oracle Certified Professional 12c and Oracle Certified Expert RAC\/GridInfra. Over time, Daniel has become increasingly interested in open source technologies, becoming \u201cTechnology Leader Open Infrastructure\u201d and PostgreSQL expert. \u00a0Based on community or EnterpriseDB tools, he develops and installs complex high available solutions with PostgreSQL. He is also a certified PostgreSQL Plus 9.0 Professional and a Postgres Advanced Server 9.4 Professional. He is a regular speaker at PostgreSQL conferences in Switzerland and Europe. Today Daniel is also supporting our customers on AWS services such as AWS RDS, database migrations into the cloud, EC2 and automated infrastructure management with AWS SSM (System Manager). He is a certified AWS Solutions Architect Professional. Prior to dbi services, Daniel was Management System Engineer at LC SYSTEMS-Engineering AG in Basel. Before that, he worked as Oracle Developper &\u00a0Project Manager at Delta Energy Solutions AG in Basel (today Powel AG). Daniel holds a diploma in Business Informatics (DHBW, Germany). His branch-related experience mainly covers the pharma industry, the financial sector, energy, lottery and telecommunications.","sameAs":["https:\/\/x.com\/westermanndanie"],"url":"https:\/\/www.dbi-services.com\/blog\/author\/daniel-westermann\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/19092","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/29"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=19092"}],"version-history":[{"count":12,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/19092\/revisions"}],"predecessor-version":[{"id":19210,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/19092\/revisions\/19210"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=19092"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=19092"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=19092"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=19092"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}