{"id":40217,"date":"2025-09-14T11:53:52","date_gmt":"2025-09-14T09:53:52","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/?p=40217"},"modified":"2025-09-14T17:29:29","modified_gmt":"2025-09-14T15:29:29","slug":"postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\/","title":{"rendered":"PostgreSQL CDC to JDBC Sink &#8211; minimal event driven architecture"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\" id=\"h-introduction\">Introduction<\/h2>\n\n\n\n<p>&#8220;Make it simple.&#8221; <br>When you have to create a solution, design an architecture, you should always take the simplest path, this is often the better solution proven by field knowledge and experiences of not scalable patterns. Which means also that you should only add the necessary and required complexity to your solutions. <br>So while following this principle I tried to look for the simplest event driven design that I could find. <br>And here it is : <br><strong>PG Source \u2192 Flink CDC \u2192 JDBC Sink \u2192 PG Data Mart<\/strong><\/p>\n\n\n\n<p><\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"277\" src=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-1024x277.png\" alt=\"\" class=\"wp-image-40218\" srcset=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-1024x277.png 1024w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-300x81.png 300w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-768x207.png 768w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image.png 1218w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p><br><strong>What each piece does<\/strong> :<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>PostgreSQL (source)<\/strong><br>Emits a change stream from the WAL using <strong>logical replication<\/strong> (<code>wal_level=logical<\/code>).<br>A <strong>publication<\/strong> defines which tables are replicated; a <strong>replication slot<\/strong> (one active reader per slot) holds WAL until the consumer confirms it.<\/li>\n\n\n\n<li><strong>Flink CDC source (Postgres)<\/strong><br>A Flink table that reads from the publication\/slot and turns WAL into a <strong>changelog stream<\/strong> (insert\/update\/delete).<br>Key options you\u2019re using:\n<ul class=\"wp-block-list\">\n<li><code>scan.incremental.snapshot.enabled=true<\/code> \u2013 non-blocking initial load<\/li>\n\n\n\n<li><code>slot.name=...<\/code> \u2013 binds the job to a specific slot<\/li>\n\n\n\n<li>primary key in the table schema \u2013 lets downstream sinks do upserts<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Flink runtime<\/strong><br>Runs a streaming job that:\n<ol class=\"wp-block-list\">\n<li><strong>Initial incremental snapshot<\/strong>: splits the table, bulk reads current rows, remembers an <strong>LSN<\/strong>.<\/li>\n\n\n\n<li><strong>Catch-up + stream<\/strong>: replays WAL from that LSN and then tails new changes.<\/li>\n\n\n\n<li><strong>Checkpoints<\/strong>: the slot LSN + operator state are stored on your S3\/MinIO path, so restarts resume exactly from the last acknowledged point.<\/li>\n<\/ol>\n<\/li>\n\n\n\n<li><strong>JDBC sink (Postgres data mart)<\/strong><br>Another Flink table. With a <strong>PRIMARY KEY<\/strong> defined, the connector performs <strong>UPSERT\/DELETE<\/strong> semantics (e.g., <code>INSERT ... ON CONFLICT DO UPDATE<\/code> in Postgres).<br>It writes in batches, flushes on checkpoints, and retries on transient errors.<\/li>\n\n\n\n<li><strong>PostgreSQL (data mart)<\/strong><br>Receives the normalized upsert\/delete stream and ends up with a 1:1 \u201ccurrent state\u201d of the source tables (ideal for BI).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-why-is-this-design-is-useful\">Why is this design is useful ?<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Near-real-time replication with transforms<\/strong>: you can filter, project, cleanse, deduplicate, join reference data, and even aggregate <strong>in Flink SQL<\/strong> before hitting the data mart\u2014something native logical replication can\u2019t do.<\/li>\n\n\n\n<li><strong>Upserts keep the mart tidy<\/strong>: the JDBC sink writes the <strong>current state<\/strong> keyed by your PKs (perfect for reporting).<\/li>\n\n\n\n<li><strong>Resilient<\/strong>: checkpoints + WAL offsets \u2192 automatic catch-up after failures\/restarts.<\/li>\n\n\n\n<li><strong>DB-friendly<\/strong>: WAL-based CDC has low OLTP impact compared to heavy ETL pulls.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-lab-demo\">LAB DEMO <\/h2>\n\n\n\n<p><strong>Architecture of the LAB : <\/strong><\/p>\n\n\n\n<p>server1 <strong>(SRC)<\/strong> \u2192 PostgreSQL 17.6 <strong>(source)<\/strong> + <strong>Flink<\/strong> (jobs run here) : <strong>IP 172.19.0.4<\/strong><\/p>\n\n\n\n<p>server2<strong> (SINKS)<\/strong> \u2192 PostgreSQL<strong> <\/strong>17.6 <strong>(data mart)<\/strong> :<strong> IP 172.20.0.4<\/strong><br><br><\/p>\n\n\n\n<p><strong>First we need data to transfer, here is a sample database that you can create yourself, additionally we also setup the PostgreSQL instance : <br><\/strong><\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\n# Execute on server1 only\nsudo -u postgres psql -c &quot;ALTER SYSTEM SET wal_level=&#039;logical&#039;;&quot;\n\n# Increase the number of concurrent replication connections allowed.\nsudo -u postgres psql -c &quot;ALTER SYSTEM SET max_wal_senders=10;&quot;\n\n# Increase the number of replication slots the server can support.\nsudo -u postgres psql -c &quot;ALTER SYSTEM SET max_replication_slots=10;&quot;\n\n# On both servers, modify the Host-Based Authentication file to allow connections from each others.\n\necho &quot;host all all 127.0.0.1\/32 trust&quot; | sudo tee -a \/etc\/postgresql\/17\/main\/pg_hba.conf\necho &quot;host all all 172.19.0.4\/32 scram-sha-256&quot; | sudo tee -a \/etc\/postgresql\/17\/main\/pg_hba.conf\necho &quot;host all all 172.20.0.4\/32 scram-sha-256&quot; | sudo tee -a \/etc\/postgresql\/17\/main\/pg_hba.conf\n\n\n# Restart the PostgreSQL service to apply all configuration changes.\nsudo systemctl restart postgresql\n\n\nsudo -u postgres createdb logistics_src\n\nsudo su - postgres\n\n# Execute a multi-statement SQL block to define and seed the schema.\npsql -U postgres -d logistics_src &amp;lt;&amp;lt;&#039;SQL&#039;\nCREATE SCHEMA logistics;\n\nCREATE TABLE logistics.customers (\n  customer_id   bigserial PRIMARY KEY,\n  name          text NOT NULL,\n  city          text,\n  email         text UNIQUE\n);\n\nCREATE TABLE logistics.products (\n  product_id    bigserial PRIMARY KEY,\n  sku           text UNIQUE NOT NULL,\n  name          text NOT NULL,\n  list_price    numeric(12,2) NOT NULL\n);\n\nCREATE TABLE logistics.orders (\n  order_id      bigserial PRIMARY KEY,\n  customer_id   bigint NOT NULL REFERENCES logistics.customers(customer_id),\n  status        text NOT NULL DEFAULT &#039;NEW&#039;,\n  order_ts      timestamptz NOT NULL DEFAULT now()\n);\n\nCREATE TABLE logistics.order_items (\n  order_id      bigint NOT NULL REFERENCES logistics.orders(order_id),\n  product_id    bigint NOT NULL REFERENCES logistics.products(product_id),\n  qty           int NOT NULL,\n  unit_price    numeric(12,2) NOT NULL,\n  PRIMARY KEY(order_id, product_id)\n);\n\nCREATE TABLE logistics.inventory (\n  product_id    bigint PRIMARY KEY REFERENCES logistics.products(product_id),\n  on_hand       int NOT NULL DEFAULT 0\n);\n\nCREATE TABLE logistics.shipments (\n  shipment_id   bigserial PRIMARY KEY,\n  order_id      bigint NOT NULL REFERENCES logistics.orders(order_id),\n  carrier       text,\n  shipped_at    timestamptz,\n  status        text\n);\n\n-- Seed initial data\nINSERT INTO logistics.customers(name,city,email)\nSELECT &#039;Customer &#039;||g, &#039;City &#039;|| (g%10), &#039;c&#039;||g||&#039;@example.com&#039;\nFROM generate_series(1,200) g;\n\nINSERT INTO logistics.products(sku,name,list_price)\nSELECT &#039;SKU-&#039;||g, &#039;Product &#039;||g, (random()*90+10)::numeric(12,2)\nFROM generate_series(1,500) g;\n\nINSERT INTO logistics.inventory(product_id,on_hand)\nSELECT product_id, (random()*100)::int\nFROM logistics.products;\n\n-- Create 100 orders\nWITH o AS (\n  INSERT INTO logistics.orders(customer_id,status)\n  SELECT (floor(random()*200)+1)::int, &#039;NEW&#039;    -- customers 1..200\n  FROM generate_series(1,100)\n  RETURNING order_id\n)\n-- For each order, choose 2 distinct products and insert items\nINSERT INTO logistics.order_items(order_id, product_id, qty, unit_price)\nSELECT o.order_id,\n       p.product_id,\n       (floor(random()*5)+1)::int AS qty,        -- qty 1..5\n       p.list_price\nFROM o\nCROSS JOIN LATERAL (\n  SELECT pr.product_id, pr.list_price\n  FROM logistics.products pr\n  ORDER BY random()\n  LIMIT 2\n) AS p;\n\nSQL\n\n\npsql -U postgres -d logistics_src -c &quot;ALTER ROLE postgres IN DATABASE logistics_src\n  SET search_path = logistics, public;&quot;\n\npostgres@LAB-CDC-SRC:~$ psql -U postgres -d logistics_src -c &quot;ALTER ROLE postgres IN DATABASE logistics_src\n  SET search_path = logistics, public;&quot;\nALTER ROLE\npostgres@LAB-CDC-SRC:~$ psql\npsql (17.6 (Ubuntu 17.6-1.pgdg24.04+1))\nType &quot;help&quot; for help.\n\npostgres=# \\l\n                                                   List of databases\n     Name      |  Owner   | Encoding | Locale Provider | Collate |  Ctype  | Locale | ICU Rules |   Access privileges\n---------------+----------+----------+-----------------+---------+---------+--------+-----------+-----------------------\n logistics_src | postgres | UTF8     | libc            | C.UTF-8 | C.UTF-8 |        |           |\n postgres      | postgres | UTF8     | libc            | C.UTF-8 | C.UTF-8 |        |           |\n template0     | postgres | UTF8     | libc            | C.UTF-8 | C.UTF-8 |        |           | =c\/postgres          +\n               |          |          |                 |         |         |        |           | postgres=CTc\/postgres\n template1     | postgres | UTF8     | libc            | C.UTF-8 | C.UTF-8 |        |           | =c\/postgres          +\n               |          |          |                 |         |         |        |           | postgres=CTc\/postgres\n(4 rows)\n\npostgres=# \\c logistics_src\nYou are now connected to database &quot;logistics_src&quot; as user &quot;postgres&quot;.\nlogistics_src=# \\dt\n             List of relations\n  Schema   |    Name     | Type  |  Owner\n-----------+-------------+-------+----------\n logistics | customers   | table | postgres\n logistics | inventory   | table | postgres\n logistics | order_items | table | postgres\n logistics | orders      | table | postgres\n logistics | products    | table | postgres\n logistics | shipments   | table | postgres\n(6 rows)\n\n\nlogistics_src=# SELECT count(*) FROM logistics.orders;\nSELECT count(*) FROM logistics.order_items;\n count\n-------\n   700\n(1 row)\n\n count\n-------\n  1100\n(1 row)\n\n\n## Here I created a slot per table on my source \npostgres=# SELECT slot_name, active, confirmed_flush_lsn FROM pg_replication_slots;\nSELECT pubname FROM pg_publication;\n       slot_name        | active | confirmed_flush_lsn\n------------------------+--------+---------------------\n flink_order_items_slot | f      | 0\/20368F0\n flink_orders_slot      | f      | 0\/20368F0\n(2 rows)\n\n pubname\n---------\n(0 rows)\n\npostgres=#\n\n<\/pre><\/div>\n\n\n<p><strong>On my target data mart I created the empty structure<\/strong> :<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nlogistics_dm=#\nSELECT count(*) FROM datamart.orders;\nSELECT count(*) FROM datamart.order_items;\n count\n-------\n     0\n(1 row)\n\n count\n-------\n     0\n(1 row)\n\nlogistics_dm=#\n<\/pre><\/div>\n\n\n<p><strong>Then we can start Flink : <br><\/strong><\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nadrien@LAB-CDC-SRC:~\/flink-1.20.2$ .\/bin\/start-cluster.sh\nStarting cluster.\nStarting standalonesession daemon on host LAB-CDC-SRC.\nStarting taskexecutor daemon on host LAB-CDC-SRC.\n<\/pre><\/div>\n\n\n<p><strong>Verify that your Flink UI is up by checking the URL http:\/\/127.0.0.1:8081 <\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"493\" src=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-1-1024x493.png\" alt=\"\" class=\"wp-image-40252\" srcset=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-1-1024x493.png 1024w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-1-300x144.png 300w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-1-768x370.png 768w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-1-1536x739.png 1536w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-1-2048x985.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>In my case I created a custom config file to handle some other aspects of my LAB like the Hudi and S3 part that you can skip. The important point in the config is more related to task manager and memory settings for the scheduler :<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nadrien@LAB-CDC-SRC:~\/flink-1.20.2$ cat conf\/flink-conf.yaml\njobmanager.rpc.address: localhost\n# Web\/API\nrest.address: 0.0.0.0\n\n# Memory (required by 1.20 to be set explicitly)\njobmanager.memory.process.size: 1200m\ntaskmanager.memory.process.size: 1600m\ntaskmanager.numberOfTaskSlots: 8\n\n# Allow multiple small jobs at once\nparallelism.default: 1     # so each job can start with 1 slot by default\njobmanager.scheduler: adaptive\n\n# Checkpoint\/savepoint locations (use MinIO so they survive restarts)\nstate.checkpoints.dir: s3a:\/\/flink\/checkpoints\nstate.savepoints.dir:   s3a:\/\/flink\/savepoints\nexecution.checkpointing.interval: 10 s\nexecution.checkpointing.mode: EXACTLY_ONCE\nexecution.checkpointing.externalized-checkpoint-retention: RETAIN_ON_CANCELLATION\n\n# Optional resilience\nrestart-strategy: fixed-delay\nrestart-strategy.fixed-delay.attempts: 10\nrestart-strategy.fixed-delay.delay: 5 s\n\n\n\n# Prefer parent loader (helps with some connector deps)\nclassloader.resolve-order: parent-first\nclassloader.check-leaked-classloader: false\n\n# S3A to MinIO (Hadoop FS)\ns3.endpoint: http:\/\/172.20.0.4:9000\ns3.path.style.access: true\ns3.access.key: admin\ns3.secret.key: adminadmin\ns3.connection.ssl.enabled: false\n  #s3.impl: org.apache.hadoop.fs.s3a.S3AFileSystem\n  #s3.aws.credentials.provider: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider\n\n\n# optional, handy for labs\nexecution.checkpointing.interval: 10 s\nenv.java.opts: --add-opens=java.base\/java.util=ALL-UNNAMED --add-opens=java.base\/java.lang=ALL-UNNAMED --add-opens=java.base\/java.lang.reflect=ALL-UNNAMED --add-opens=java.base\/sun.nio.ch=ALL-UNNAMED --add-opens=java.base\/java.nio=ALL-UNNAMED -Djdk.attach.allowAttachSelf=true\nadrien@LAB-CDC-SRC:~\/flink-1.20.2$\n<\/pre><\/div>\n\n\n<p><strong>Then we start the SQL client, <\/strong><\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nadrien@LAB-CDC-SRC:~\/flink-1.20.2$ .\/bin\/sql-client.sh\n\n                                   \u2592\u2593\u2588\u2588\u2593\u2588\u2588\u2592\n                               \u2593\u2588\u2588\u2588\u2588\u2592\u2592\u2588\u2593\u2592\u2593\u2588\u2588\u2588\u2593\u2592\n                            \u2593\u2588\u2588\u2588\u2593\u2591\u2591        \u2592\u2592\u2592\u2593\u2588\u2588\u2592  \u2592\n                          \u2591\u2588\u2588\u2592   \u2592\u2592\u2593\u2593\u2588\u2593\u2593\u2592\u2591      \u2592\u2588\u2588\u2588\u2588\n                          \u2588\u2588\u2592         \u2591\u2592\u2593\u2588\u2588\u2588\u2592    \u2592\u2588\u2592\u2588\u2592\n                            \u2591\u2593\u2588            \u2588\u2588\u2588   \u2593\u2591\u2592\u2588\u2588\n                              \u2593\u2588       \u2592\u2592\u2592\u2592\u2592\u2593\u2588\u2588\u2593\u2591\u2592\u2591\u2593\u2593\u2588\n                            \u2588\u2591 \u2588   \u2592\u2592\u2591       \u2588\u2588\u2588\u2593\u2593\u2588 \u2592\u2588\u2592\u2592\u2592\n                            \u2588\u2588\u2588\u2588\u2591   \u2592\u2593\u2588\u2593      \u2588\u2588\u2592\u2592\u2592 \u2593\u2588\u2588\u2588\u2592\n                         \u2591\u2592\u2588\u2593\u2593\u2588\u2588       \u2593\u2588\u2592    \u2593\u2588\u2592\u2593\u2588\u2588\u2593 \u2591\u2588\u2591\n                   \u2593\u2591\u2592\u2593\u2588\u2588\u2588\u2588\u2592 \u2588\u2588         \u2592\u2588    \u2588\u2593\u2591\u2592\u2588\u2592\u2591\u2592\u2588\u2592\n                  \u2588\u2588\u2588\u2593\u2591\u2588\u2588\u2593  \u2593\u2588           \u2588   \u2588\u2593 \u2592\u2593\u2588\u2593\u2593\u2588\u2592\n                \u2591\u2588\u2588\u2593  \u2591\u2588\u2591            \u2588  \u2588\u2592 \u2592\u2588\u2588\u2588\u2588\u2588\u2593\u2592 \u2588\u2588\u2593\u2591\u2592\n               \u2588\u2588\u2588\u2591 \u2591 \u2588\u2591          \u2593 \u2591\u2588 \u2588\u2588\u2588\u2588\u2588\u2592\u2591\u2591    \u2591\u2588\u2591\u2593  \u2593\u2591\n              \u2588\u2588\u2593\u2588 \u2592\u2592\u2593\u2592          \u2593\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2593\u2591       \u2592\u2588\u2592 \u2592\u2593 \u2593\u2588\u2588\u2593\n           \u2592\u2588\u2588\u2593 \u2593\u2588 \u2588\u2593\u2588       \u2591\u2592\u2588\u2588\u2588\u2588\u2588\u2593\u2593\u2592\u2591         \u2588\u2588\u2592\u2592  \u2588 \u2592  \u2593\u2588\u2592\n           \u2593\u2588\u2593  \u2593\u2588 \u2588\u2588\u2593 \u2591\u2593\u2593\u2593\u2593\u2593\u2593\u2593\u2592              \u2592\u2588\u2588\u2593           \u2591\u2588\u2592\n           \u2593\u2588    \u2588 \u2593\u2588\u2588\u2588\u2593\u2592\u2591              \u2591\u2593\u2593\u2593\u2588\u2588\u2588\u2593          \u2591\u2592\u2591 \u2593\u2588\n           \u2588\u2588\u2593    \u2588\u2588\u2592    \u2591\u2592\u2593\u2593\u2588\u2588\u2588\u2593\u2593\u2593\u2593\u2593\u2588\u2588\u2588\u2588\u2588\u2588\u2593\u2592            \u2593\u2588\u2588\u2588  \u2588\n          \u2593\u2588\u2588\u2588\u2592 \u2588\u2588\u2588   \u2591\u2593\u2593\u2592\u2591\u2591   \u2591\u2593\u2588\u2588\u2588\u2588\u2593\u2591                  \u2591\u2592\u2593\u2592  \u2588\u2593\n          \u2588\u2593\u2592\u2592\u2593\u2593\u2588\u2588  \u2591\u2592\u2592\u2591\u2591\u2591\u2592\u2592\u2592\u2592\u2593\u2588\u2588\u2593\u2591                            \u2588\u2593\n          \u2588\u2588 \u2593\u2591\u2592\u2588   \u2593\u2593\u2593\u2593\u2592\u2591\u2591  \u2592\u2588\u2593       \u2592\u2593\u2593\u2588\u2588\u2593    \u2593\u2592          \u2592\u2592\u2593\n          \u2593\u2588\u2593 \u2593\u2592\u2588  \u2588\u2593\u2591  \u2591\u2592\u2593\u2593\u2588\u2588\u2592            \u2591\u2593\u2588\u2592   \u2592\u2592\u2592\u2591\u2592\u2592\u2593\u2588\u2588\u2588\u2588\u2588\u2592\n           \u2588\u2588\u2591 \u2593\u2588\u2592\u2588\u2592  \u2592\u2593\u2593\u2592  \u2593\u2588                \u2588\u2591      \u2591\u2591\u2591\u2591   \u2591\u2588\u2592\n           \u2593\u2588   \u2592\u2588\u2593   \u2591     \u2588\u2591                \u2592\u2588              \u2588\u2593\n            \u2588\u2593   \u2588\u2588         \u2588\u2591                 \u2593\u2593        \u2592\u2588\u2593\u2593\u2593\u2592\u2588\u2591\n             \u2588\u2593 \u2591\u2593\u2588\u2588\u2591       \u2593\u2592                  \u2593\u2588\u2593\u2592\u2591\u2591\u2591\u2592\u2593\u2588\u2591    \u2592\u2588\n              \u2588\u2588   \u2593\u2588\u2593\u2591      \u2592                    \u2591\u2592\u2588\u2592\u2588\u2588\u2592      \u2593\u2593\n               \u2593\u2588\u2592   \u2592\u2588\u2593\u2592\u2591                         \u2592\u2592 \u2588\u2592\u2588\u2593\u2592\u2592\u2591\u2591\u2592\u2588\u2588\n                \u2591\u2588\u2588\u2592    \u2592\u2593\u2593\u2592                     \u2593\u2588\u2588\u2593\u2592\u2588\u2592 \u2591\u2593\u2593\u2593\u2593\u2592\u2588\u2593\n                  \u2591\u2593\u2588\u2588\u2592                          \u2593\u2591  \u2592\u2588\u2593\u2588  \u2591\u2591\u2592\u2592\u2592\n                      \u2592\u2593\u2593\u2593\u2593\u2593\u2592\u2592\u2592\u2592\u2592\u2592\u2592\u2592\u2592\u2592\u2592\u2592\u2592\u2592\u2592\u2592\u2592\u2592\u2592\u2592\u2592\u2592\u2592\u2591\u2591\u2593\u2593  \u2593\u2591\u2592\u2588\u2591\n\n    ______ _ _       _       _____  ____  _         _____ _ _            _  BETA\n   |  ____| (_)     | |     \/ ____|\/ __ \\| |       \/ ____| (_)          | |\n   | |__  | |_ _ __ | | __ | (___ | |  | | |      | |    | |_  ___ _ __ | |_\n   |  __| | | | &#039;_ \\| |\/ \/  \\___ \\| |  | | |      | |    | | |\/ _ \\ &#039;_ \\| __|\n   | |    | | | | | |   &amp;lt;   ____) | |__| | |____  | |____| | |  __\/ | | | |_\n   |_|    |_|_|_| |_|_|\\_\\ |_____\/ \\___\\_\\______|  \\_____|_|_|\\___|_| |_|\\__|\n\n        Welcome! Enter &#039;HELP;&#039; to list all available commands. &#039;QUIT;&#039; to exit.\n\nCommand history file path: \/home\/adrien\/.flink-sql-history\n\n\n\n\n\n-- Set the checkpointing interval for the SQL client session.\n-- While configured globally in flink-conf.yaml, setting it here ensures it applies.\nSET &#039;execution.checkpointing.interval&#039; = &#039;10s&#039;;\n\n-- =============================================================================\n-- DEFINE CDC SOURCE TABLES\n-- =============================================================================\n\n-- This table definition maps to the &#039;logistics.orders&#039; table in PostgreSQL.\n-- The &#039;postgres-cdc&#039; connector is used to stream changes.\nSET &#039;execution.checkpointing.interval&#039; = &#039;10s&#039;;\n\nCREATE TABLE src_orders (\n  order_id BIGINT,\n  customer_id BIGINT,\n  status STRING,\n  order_ts TIMESTAMP(3),\n  PRIMARY KEY (order_id) NOT ENFORCED\n) WITH (\n  &#039;connector&#039; = &#039;postgres-cdc&#039;,\n  &#039;hostname&#039; = &#039;172.19.0.4&#039;,\n  &#039;port&#039; = &#039;5432&#039;,\n  &#039;username&#039; = &#039;postgres&#039;,\n  &#039;password&#039; = &#039;your_postgres_password&#039;,\n  &#039;database-name&#039; = &#039;logistics_src&#039;,\n  &#039;schema-name&#039; = &#039;logistics&#039;,\n  &#039;table-name&#039;  = &#039;orders&#039;,\n  &#039;slot.name&#039;   = &#039;flink_orders_slot&#039;,\n  &#039;decoding.plugin.name&#039; = &#039;pgoutput&#039;,\n  &#039;scan.incremental.snapshot.enabled&#039; = &#039;true&#039;\n);\n\nCREATE TABLE src_order_items (\n  order_id BIGINT,\n  product_id BIGINT,\n  qty INT,\n  unit_price DECIMAL(12,2),\n  PRIMARY KEY (order_id, product_id) NOT ENFORCED\n) WITH (\n  &#039;connector&#039; = &#039;postgres-cdc&#039;,\n  &#039;hostname&#039; = &#039;172.19.0.4&#039;,\n  &#039;port&#039; = &#039;5432&#039;,\n  &#039;username&#039; = &#039;postgres&#039;,\n  &#039;password&#039; = &#039;your_postgres_password&#039;,\n  &#039;database-name&#039; = &#039;logistics_src&#039;,\n  &#039;schema-name&#039; = &#039;logistics&#039;,\n  &#039;table-name&#039;  = &#039;order_items&#039;,\n  &#039;slot.name&#039;   = &#039;flink_order_items_slot&#039;,\n  &#039;decoding.plugin.name&#039; = &#039;pgoutput&#039;,\n  &#039;scan.incremental.snapshot.enabled&#039; = &#039;true&#039;\n);\n\nCREATE TABLE dm_orders (\n  order_id BIGINT,\n  customer_id BIGINT,\n  status STRING,\n  order_ts TIMESTAMP(3),\n  PRIMARY KEY (order_id) NOT ENFORCED\n) WITH (\n  &#039;connector&#039; = &#039;jdbc&#039;,\n  &#039;url&#039; = &#039;jdbc:postgresql:\/\/172.20.0.4:5432\/logistics_dm&#039;,\n  &#039;table-name&#039; = &#039;datamart.orders&#039;,\n  &#039;username&#039; = &#039;postgres&#039;,\n  &#039;password&#039; = &#039;your_postgres_password&#039;,\n  &#039;driver&#039;    = &#039;org.postgresql.Driver&#039;\n);\n\nCREATE TABLE dm_order_items (\n  order_id BIGINT,\n  product_id BIGINT,\n  qty INT,\n  unit_price DECIMAL(12,2),\n  PRIMARY KEY (order_id, product_id) NOT ENFORCED\n) WITH (\n  &#039;connector&#039; = &#039;jdbc&#039;,\n  &#039;url&#039; = &#039;jdbc:postgresql:\/\/172.20.0.4:5432\/logistics_dm&#039;,\n  &#039;table-name&#039; = &#039;datamart.order_items&#039;,\n  &#039;username&#039; = &#039;postgres&#039;,\n  &#039;password&#039; = &#039;your_postgres_password&#039;,\n  &#039;driver&#039;    = &#039;org.postgresql.Driver&#039;\n);\n\n\n\n\nINSERT INTO dm_orders          SELECT * FROM src_orders;\nINSERT INTO dm_order_items     SELECT * FROM src_order_items;\n<\/pre><\/div>\n\n\n<p>So here we just declared the source tables and the data mart tables, note the connector type. <br>Once the tables are declare in Flink you can then start the pipeline with the INSERT INTO statements. <br><br>At the end you should get something like this in the sql_client.sh : <br><\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nFlink SQL&amp;gt; &#x5B;INFO] Submitting SQL update statement to the cluster...\n&#x5B;INFO] SQL update statement has been successfully submitted to the cluster:\nJob ID: 1e776162f4e23447ed5a9546ff464a61\n\n\nFlink SQL&amp;gt; &#x5B;INFO] Submitting SQL update statement to the cluster...\n&#x5B;INFO] SQL update statement has been successfully submitted to the cluster:\nJob ID: 14ea0afbf7e2fda44efc968c189ee480\n<\/pre><\/div>\n\n\n<p>And in Flink : <\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"396\" src=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-2-1024x396.png\" alt=\"\" class=\"wp-image-40255\" srcset=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-2-1024x396.png 1024w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-2-300x116.png 300w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-2-768x297.png 768w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-2-1536x593.png 1536w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-2-2048x791.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"527\" src=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-3-1024x527.png\" alt=\"\" class=\"wp-image-40256\" srcset=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-3-1024x527.png 1024w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-3-300x154.png 300w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-3-768x395.png 768w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-3-1536x790.png 1536w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/image-3-2048x1054.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Then verify that on the target data mart the tables are now populated : <\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\npostgres=# \\c logistics_dm\nYou are now connected to database &quot;logistics_dm&quot; as user &quot;postgres&quot;.\nlogistics_dm=#\nSELECT count(*) FROM datamart.orders;\nSELECT count(*) FROM datamart.order_items;\n count\n-------\n   700\n(1 row)\n\n count\n-------\n  1100\n(1 row)\n\nlogistics_dm=# select * from order_items limit 5;\n order_id | product_id | qty | unit_price | discount\n----------+------------+-----+------------+----------\n      224 |        223 |   4 |      15.44 |     0.00\n       96 |         94 |   1 |      53.91 |     0.00\n      689 |        143 |   3 |      70.71 |     0.00\n      290 |        223 |   3 |      15.44 |     0.00\n       93 |        191 |   2 |      95.85 |     0.00\n(5 rows)\n\nlogistics_dm=# select * from orders limit 5;\n order_id | customer_id | status |           order_ts            | channel\n----------+-------------+--------+-------------------------------+---------\n      310 |         163 | NEW    | 2025-09-09 14:36:47.247234+00 |\n      305 |          69 | NEW    | 2025-09-09 14:36:47.247234+00 |\n      304 |          14 | NEW    | 2025-09-09 14:36:47.247234+00 |\n      303 |         122 | NEW    | 2025-09-09 14:36:47.247234+00 |\n      302 |           9 | NEW    | 2025-09-09 14:36:47.247234+00 |\n(5 rows)\n<\/pre><\/div>\n\n\n<p>And this is it, we did the setup of the minimal event-driven architecture.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-end-to-end-flow-what-happens-when-you-insert-into-dm-select-from-src\">End-to-end flow (what happens when you \u201cINSERT INTO dm_* SELECT * FROM src_*\u201d)<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Flink CDC attaches to the slot<\/strong> and (if configured) creates\/uses the publication.<\/li>\n\n\n\n<li>It takes a <strong>consistent snapshot<\/strong> of <code>orders<\/code>\/<code>order_items<\/code>, while buffering concurrent WAL changes.<\/li>\n\n\n\n<li>Once snapshot is done, it <strong>emits<\/strong> the buffered changes and continues <strong>streaming<\/strong> new WAL records.<\/li>\n\n\n\n<li>The <strong>JDBC sink<\/strong> receives a relational changelog:\n<ul class=\"wp-block-list\">\n<li><code>INSERT<\/code> \u2192 <code>INSERT \u2026 ON CONFLICT DO UPDATE<\/code> (upsert)<\/li>\n\n\n\n<li><code>UPDATE<\/code> \u2192 treated as upsert<\/li>\n\n\n\n<li><code>DELETE<\/code> \u2192 <code>DELETE \u2026 WHERE pk = ?<\/code><\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Checkpoints<\/strong> coordinate the CDC source offsets with the sink flush so recovery doesn\u2019t duplicate effects. With upserts, the pipeline is <strong>effectively-once<\/strong> even if a retry happens.<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-operational-knobs-that-matter\">Operational knobs that matter<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>One slot per independent reader<\/strong> (or a single job fan-out via Statement Set).<\/li>\n\n\n\n<li><strong>Checkpointing<\/strong> (you\u2019ve set it):<br><code>execution.checkpointing.interval=10 s<\/code> + S3 paths for <code>state.checkpoints.dir<\/code>\/<code>state.savepoints.dir<\/code>.<\/li>\n\n\n\n<li><strong>Target DDL<\/strong>: create the DM tables up front with <strong>PRIMARY KEY<\/strong> to enable upserts.<\/li>\n\n\n\n<li><strong>Throughput<\/strong>: increase job\/connector <strong>parallelism<\/strong>, adjust JDBC sink batch size\/interval if needed (defaults are usually fine for this lab).<\/li>\n\n\n\n<li><strong>DDL changes on the source<\/strong>: Postgres logical decoding does <strong>not<\/strong> emit DDL \u2192 when schemas change, <strong>redeploy the Flink table DDL<\/strong> (and adjust the target tables). Use Liquibase\/Flyway to manage that cleanly.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-limitations-to-keep-in-mind\">Limitations to keep in mind<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>PK required<\/strong> for clean upserts\/deletes. Without it, the JDBC sink would just append.<\/li>\n\n\n\n<li>Pros (vs. PostgreSQL logical replication)<\/li>\n\n\n\n<li><strong>DDL<\/strong>: manual coordination (expand\/contract) is required.<\/li>\n\n\n\n<li><strong>Slots<\/strong>: one active consumer per slot; multiple slots can increase WAL retention if a reader lags or is stopped.<\/li>\n\n\n\n<li><strong>Backpressure<\/strong>: a slow data mart will throttle the job; watch Flink backpressure metrics and tune JDBC batching\/parallelism.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-pros-of-this-design\">Pros of this design<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Transform while you move<\/strong>: filter, project, join, enrich, aggregate, de-duplicate, derive SCD logic\u2014before data lands in the target.<\/li>\n\n\n\n<li><strong>Fan-out<\/strong>: the same CDC stream can drive <strong>many<\/strong> targets (multiple PG DBs, ClickHouse, Elasticsearch, etc.).<\/li>\n\n\n\n<li><strong>Decoupling &amp; safety<\/strong>: backpressure, retries, checkpointed state, and rate-limiting protect the source and the target; you can shape load to the DM.<\/li>\n\n\n\n<li><strong>Schema mediation<\/strong>: implement expand\/contract, rename mapping, default values\u2014logical replication needs the same schema on both sides.<\/li>\n\n\n\n<li><strong>Non-PG targets<\/strong>: works when your sink isn\u2019t Postgres (or not only Postgres).<\/li>\n\n\n\n<li><strong>Observability<\/strong>: Flink UI and metrics on lag, checkpoints, failures, couple it with already proven Prometheus exporters. <\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-cons\">Cons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>More moving parts<\/strong>: Flink cluster + connectors to operate.<\/li>\n\n\n\n<li><strong>Latency a bit higher<\/strong>: seconds (checkpoint cadence, batching) instead of sub-second WAL apply.<\/li>\n\n\n\n<li><strong>DDL isn\u2019t automatic<\/strong>: Postgres logical decoding doesn\u2019t emit DDL; you still manage schema changes (expand\/contract + redeploy).<\/li>\n\n\n\n<li><strong>Throughput tuning<\/strong>: JDBC upserts can bottleneck a single DB; you tune parallelism and flush\/batch settings.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">When <strong>logical replication<\/strong> is better<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>You just need a <strong>near-real-time 1:1 copy<\/strong> PG\u2192PG, <strong>no transforms<\/strong>, same schema, and <strong>lowest latency<\/strong> with minimal ops.<\/li>\n\n\n\n<li>You\u2019re okay with subscriber-side work (indexes\/views) and whole-table subscription (no row-level filtering).<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-when-should-you-upgrade-with-a-hudi-sink-data-lake\">When should you \u201cupgrade\u201d with a <strong>Hudi sink<\/strong> (data lake)<\/h3>\n\n\n\n<p>Add Hudi when you need things a database replica can\u2019t give you:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>History &amp; replayability<\/strong>: keep the raw truth (Bronze) cheaply; rebuild downstream tables any time.<\/li>\n\n\n\n<li><strong>Upserts\/deletes at scale on files<\/strong>: CDC-friendly <strong>MOR\/COW<\/strong> tables.<\/li>\n\n\n\n<li><strong>Time travel &amp; incremental pulls<\/strong>: audits, backtests, point-in-time reprocessing.<\/li>\n\n\n\n<li><strong>Many heterogeneous consumers<\/strong>: BI + ML + ad-hoc engines (Trino\/Spark\/Presto) without re-extracting from OLTP.<\/li>\n\n\n\n<li><strong>Big volumes<\/strong>: storage and compute scale independently; you can compaction\/cluster off-peak.<\/li>\n\n\n\n<li>Scale compute and storage independently.<\/li>\n<\/ul>\n\n\n\n<p><strong>Trade-off:<\/strong> more infra (S3\/MinIO + compaction), higher \u201ccold\u201d query latency than a hot DB, and you still materialize serving tables for apps.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-what-i-don-t-like-with-this-design\"><strong>What I don&#8217;t like with this design : <\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Java based <\/strong>: I don&#8217;t like to handle Java issues, runtime and versions. I am not a developer and those kinds of this should be packaged in a way that it makes them run fast and easy. This open source version and setup is not that user-friendly and requires deep knowledge of the tool, but realistically not that much more than any other tool that would do similar things. Additionally, if your are going to run this in production there is a high likelihood that your are going to use Kubernetes which is going to solve those issues and offer scalability. <\/li>\n\n\n\n<li><strong>The versions dependencies <\/strong>: I did figure out the hard way that not all latest version of all packages used are compatible with each other. In the open source world some projects sometimes need to catch up in development. Here I need to use Flink 1.20.2 to have CDC working with HUDI for example and because I want to use both JDBC connector and HUDI Sink I had to downgrade the entire stack. So be careful Flink 1.20 is the LTS version so this is fine but if you want to use the latest feature of the stable version you might want to check that first. <\/li>\n\n\n\n<li><strong>Cloud <\/strong>: this operational complexity for setup and management is handle on the cloud hence the strong argument to go for it appart from the obvious data sensitivity issue but that last part is more a labelling and classification issue than really a Flink usage argument. If your company is using not taking leverage of cloud solution because it is afraid of data loss, this is a high indicator of a lack of maturity in that area rather than a technical limitation. <\/li>\n<\/ul>\n\n\n\n<p><strong>Additional notes<\/strong> : this setup is not meant for production, this is a simple showcase for lab purpose that you can easily reproduce, here there is not persistence (if you restart the Flink processes your pipeline are lost), if you want a production ready setup refer to the official documentation and look for Kubernetes installations probably : <a href=\"https:\/\/nightlies.apache.org\/flink\/flink-cdc-docs-stable\/docs\/get-started\/introduction\/\">Introduction | Apache Flink CDC<\/a>.<\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction &#8220;Make it simple.&#8221; When you have to create a solution, design an architecture, you should always take the simplest path, this is often the better solution proven by field knowledge and experiences of not scalable patterns. Which means also that you should only add the necessary and required complexity to your solutions. So while [&hellip;]<\/p>\n","protected":false},"author":153,"featured_media":40270,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[83],"tags":[614,3666,3667,2602],"type_dbi":[],"class_list":["post-40217","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-postgresql","tag-cdc","tag-event-driven","tag-flink","tag-postgresql-2"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.2 (Yoast SEO v27.4) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>PostgreSQL CDC to JDBC Sink - minimal event driven architecture - dbi Blog<\/title>\n<meta name=\"description\" content=\"When a data lake is to much but event-driven pipelines are required. I tried this Flink CDC + JDBC connector design.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"PostgreSQL CDC to JDBC Sink - minimal event driven architecture\" \/>\n<meta property=\"og:description\" content=\"When a data lake is to much but event-driven pipelines are required. I tried this Flink CDC + JDBC connector design.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2025-09-14T09:53:52+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-09-14T15:29:29+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/elephant.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1344\" \/>\n\t<meta property=\"og:image:height\" content=\"768\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Adrien Obernesser\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Adrien Obernesser\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\\\/\"},\"author\":{\"name\":\"Adrien Obernesser\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/fd2ab917212ce0200c7618afaa7fdbcd\"},\"headline\":\"PostgreSQL CDC to JDBC Sink &#8211; minimal event driven architecture\",\"datePublished\":\"2025-09-14T09:53:52+00:00\",\"dateModified\":\"2025-09-14T15:29:29+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\\\/\"},\"wordCount\":1481,\"commentCount\":0,\"image\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2025\\\/09\\\/elephant.png\",\"keywords\":[\"cdc\",\"Event-driven\",\"Flink\",\"postgresql\"],\"articleSection\":[\"PostgreSQL\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\\\/\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\\\/\",\"name\":\"PostgreSQL CDC to JDBC Sink - minimal event driven architecture - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2025\\\/09\\\/elephant.png\",\"datePublished\":\"2025-09-14T09:53:52+00:00\",\"dateModified\":\"2025-09-14T15:29:29+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/fd2ab917212ce0200c7618afaa7fdbcd\"},\"description\":\"When a data lake is to much but event-driven pipelines are required. I tried this Flink CDC + JDBC connector design.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2025\\\/09\\\/elephant.png\",\"contentUrl\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2025\\\/09\\\/elephant.png\",\"width\":1344,\"height\":768},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"PostgreSQL CDC to JDBC Sink &#8211; minimal event driven architecture\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/fd2ab917212ce0200c7618afaa7fdbcd\",\"name\":\"Adrien Obernesser\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/dc9316c729e50107159e0a1e631b9c1742ce8898576887d0103c83b1ca3bc9e6?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/dc9316c729e50107159e0a1e631b9c1742ce8898576887d0103c83b1ca3bc9e6?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/dc9316c729e50107159e0a1e631b9c1742ce8898576887d0103c83b1ca3bc9e6?s=96&d=mm&r=g\",\"caption\":\"Adrien Obernesser\"},\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/author\\\/adrienobernesser\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"PostgreSQL CDC to JDBC Sink - minimal event driven architecture - dbi Blog","description":"When a data lake is to much but event-driven pipelines are required. I tried this Flink CDC + JDBC connector design.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\/","og_locale":"en_US","og_type":"article","og_title":"PostgreSQL CDC to JDBC Sink - minimal event driven architecture","og_description":"When a data lake is to much but event-driven pipelines are required. I tried this Flink CDC + JDBC connector design.","og_url":"https:\/\/www.dbi-services.com\/blog\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\/","og_site_name":"dbi Blog","article_published_time":"2025-09-14T09:53:52+00:00","article_modified_time":"2025-09-14T15:29:29+00:00","og_image":[{"width":1344,"height":768,"url":"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/elephant.png","type":"image\/png"}],"author":"Adrien Obernesser","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Adrien Obernesser","Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\/"},"author":{"name":"Adrien Obernesser","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/fd2ab917212ce0200c7618afaa7fdbcd"},"headline":"PostgreSQL CDC to JDBC Sink &#8211; minimal event driven architecture","datePublished":"2025-09-14T09:53:52+00:00","dateModified":"2025-09-14T15:29:29+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\/"},"wordCount":1481,"commentCount":0,"image":{"@id":"https:\/\/www.dbi-services.com\/blog\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\/#primaryimage"},"thumbnailUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/elephant.png","keywords":["cdc","Event-driven","Flink","postgresql"],"articleSection":["PostgreSQL"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\/","url":"https:\/\/www.dbi-services.com\/blog\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\/","name":"PostgreSQL CDC to JDBC Sink - minimal event driven architecture - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\/#primaryimage"},"image":{"@id":"https:\/\/www.dbi-services.com\/blog\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\/#primaryimage"},"thumbnailUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/elephant.png","datePublished":"2025-09-14T09:53:52+00:00","dateModified":"2025-09-14T15:29:29+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/fd2ab917212ce0200c7618afaa7fdbcd"},"description":"When a data lake is to much but event-driven pipelines are required. I tried this Flink CDC + JDBC connector design.","breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.dbi-services.com\/blog\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\/#primaryimage","url":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/elephant.png","contentUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2025\/09\/elephant.png","width":1344,"height":768},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/postgresql-cdc-to-jdbc-sink-minimal-event-driven-architecture\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"PostgreSQL CDC to JDBC Sink &#8211; minimal event driven architecture"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/fd2ab917212ce0200c7618afaa7fdbcd","name":"Adrien Obernesser","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/dc9316c729e50107159e0a1e631b9c1742ce8898576887d0103c83b1ca3bc9e6?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/dc9316c729e50107159e0a1e631b9c1742ce8898576887d0103c83b1ca3bc9e6?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/dc9316c729e50107159e0a1e631b9c1742ce8898576887d0103c83b1ca3bc9e6?s=96&d=mm&r=g","caption":"Adrien Obernesser"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/adrienobernesser\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/40217","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/153"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=40217"}],"version-history":[{"count":29,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/40217\/revisions"}],"predecessor-version":[{"id":40277,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/40217\/revisions\/40277"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media\/40270"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=40217"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=40217"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=40217"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=40217"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}