Every MariaDB (and MySQL) production DBA has felt the pain of the classic binlog performance tax or replication management. Every committed transaction must cross a two-phase commit (2PC) boundary between InnoDB and the binary log — two separate, sequential fsync() calls per transaction group, regardless of what innodb_flush_log_at_trx_commit is set to. The binlog has always been a flat file, written independently of InnoDB’s Write-Ahead Log (WAL), which means:

• Two durability paths to synchronize on every commit
• Complex crash recovery logic to reconcile InnoDB state with binlog state
• sync_binlog=1 + innodb_flush_log_at_trx_commit=1 = two fsyncs, always
• Performance degrades steeply under high concurrency, especially on cloud disks where fsync latency is measured in milliseconds

MariaDB 12.3 ships a fundamental architectural answer to this problem, tracked under MDEV-34705 and authored by Kristian Nielsen: the binary log is now stored inside InnoDB tablespaces, using InnoDB’s own Write-Ahead Log for durability. This is not an incremental tuning — it is a redesign of the commit pipeline.

To understand what changed, you need to understand what the classic binlog commit path looks like:

BEGIN transaction
  → InnoDB: write undo log (redo log)
  → InnoDB: prepare (XA prepare, writes to redo log)
  → Binlog: write event to binlog file
  → Binlog: fsync() [sync_binlog=1]
  → InnoDB: commit (writes commit record to redo log)
  → InnoDB: fsync() [innodb_flush_log_at_trx_commit=1]
COMMIT

The problem: two separate fsync() calls, two separate file paths, and InnoDB cannot commit without first knowing that the binlog has been durably written. If the server crashes between the binlog write and the InnoDB commit, recovery must scan the binlog to find prepared-but-not-committed transactions. This is expensive both in steady-state (latency) and at recovery time (scan time).

Group commit mitigates this partially — multiple transactions can be flushed together — but the fundamental 2PC overhead remains.

With binlog_storage_engine=innodb, the binlog is no longer a separate flat file. It lives inside the InnoDB storage engine, in InnoDB tablespace files with the .ibb extension, pre-allocated at max_binlog_size (default 1 GB each). The commit path becomes:

BEGIN transaction
  → InnoDB: write undo log + binlog event (single redo log)
  → InnoDB: commit (one fsync if innodb_flush_log_at_trx_commit=1)
COMMIT

The two-phase commit between binlog and engine is eliminated. The binlog and data changes are atomic — they land or fail together in the InnoDB WAL.

This has two dramatic consequences:

1. innodb_flush_log_at_trx_commit=0 or =2 becomes safe. Because crash recovery is now handled entirely by InnoDB, you can run with innodb_flush_log_at_trx_commit=0 (no fsync, OS buffer only) and still have a consistent binlog after a crash. Previously, this setting was dangerous for replication because the binlog and InnoDB could diverge after a crash. With the new model, they cannot diverge — they are the same file.

2. With innodb_flush_log_at_trx_commit=1, one fsync replaces two. The most common production setting now only costs a single WAL fsync. Group commit opportunities also improve because the binlog is no longer a separate bottleneck.

(You might want to check the documentation for this parameter link.)

New binlog files use the .ibb extension and are pre-allocated. Here is what the binlog directory looks like side by side:

Classic FILE binlog:

$ ls -lah /var/lib/mysql/binlog/
drwx------ 2 mysql mysql 4.0K  #binlog_cache_files
-rw-rw---- 1 mysql mysql 1.9K  mysqld-bin.000001
-rw-rw---- 1 mysql mysql 4.0K  mysqld-bin.000001.idx
-rw-rw---- 1 mysql mysql 1.4K  mysqld-bin.000002
-rw-rw---- 1 mysql mysql    0  mysqld-bin.000002.idx
-rw-rw---- 1 mysql mysql   80  mysqld-bin.index

InnoDB binlog:

$ ls -lah /var/lib/mysql/binlog/
-rw-rw---- 1 mysql mysql 1.0G  binlog-000000.ibb
-rw-rw---- 1 mysql mysql 1.0G  binlog-000001.ibb

The difference is immediately striking: the .ibb files are pre-allocated at max_binlog_size (default 1 GB). There is no .index file, no .idx GTID index files, no .state file. GTID state is periodically written as state records inside the binlog itself, controlled by --innodb-binlog-state-interval (bytes between state records). GTID position recovery scans from the last state record forward.

Another subtle difference visible in the mariadb-binlog output: all InnoDB binlog events show end_log_pos 0 (position tracking is handled by InnoDB internally), whereas FILE binlog events show actual byte offsets (end_log_pos 257, end_log_pos 330, etc.).

Events can span multiple .ibb files, and parts of the same event can be interleaved across files. mariadb-binlog coalesces them transparently. For correct output across files, pass all files at once in order.

This is a significant improvement over the old model: mariadb-backup now includes the binlog in the backup by default, transactionally consistent with the data. The backed-up server is not blocked during binlog copy. A restored backup can be turned into a replica using CHANGE MASTER ... MASTER_DEMOTE_TO_SLAVE=1 directly — no separate binlog position reconciliation needed.

Operational Impact: GTID, Replica Resync, and Split-Brain Recovery

This is arguably the most impactful day-to-day change for DBAs apart from the TPS throughput. You can’t use anything else than GTID mode for replication if you use innodb as the binlog storage engine. I still run into a lot of MariaDB 10.11 so some DBAs might not use this on a daily basis yet. To help them understand the next part is a quick reminder of the GTID capabilities that has been around for 12 years already…

What MASTER_USE_GTID = slave_pos Actually Means

A common point of confusion when reading MariaDB documentation for the first time: in the CHANGE MASTER TO statement, slave_pos looks like a placeholder you are supposed to fill in. It is not. It is a literal enum keyword — one of exactly three accepted values:

MASTER_USE_GTID = no           -- classic file/offset mode, GTID disabled
MASTER_USE_GTID = slave_pos    -- use @@gtid_slave_pos (what this replica last received)
MASTER_USE_GTID = current_pos  -- use @@gtid_current_pos (slave_pos + any local writes)

slave_pos is not a value you provide — it is an instruction telling MariaDB to read the starting replication position from the server’s own @@gtid_slave_pos system variable, which was populated automatically during the backup restore. You are providing an instruction, not data. The actual position is determined by the engine, not by you.

For completeness: MariaDB accepts the keyword unquoted (unlike MySQL’s tendency to quote string enums), which is why it looks like a variable name in documentation examples.

Replica Resync Before and After: The Concrete Difference

This is best understood by comparing the full procedure side by side.

With FILE binlog (classic):

Problem: two independent sources of truth to reconcile.

1. Take fresh backup with mariadb-backup
2. Read xtrabackup_binlog_info from the backup:
   → binlog.000042 position 198472910
   BUT: this position may not match what InnoDB actually committed
   (MDEV-21611 — mariabackup does not always update the InnoDB
   binlog position during prepare, causing mismatch)
3. Restore the backup on the replica
4. Manually declare:
   CHANGE MASTER TO
   MASTER_LOG_FILE = ‘binlog.000042’, ← you looked this up
   MASTER_LOG_POS = 198472910; ← you looked this up
5. START SLAVE;
6. Watch for errors, validate row counts, hope the position was right

The fragility: step 2/4 requires you to provide specific data (a filename string and a byte offset integer). If that data is wrong by even one transaction — which MDEV-21611 demonstrates can happen — the replica either re-applies events it already has, or misses events entirely. This produces silent data drift, not an immediate error.

With InnoDB binlog + GTID:

Problem: eliminated. One source of truth.

1. Take fresh backup with mariadb-backup
   → .ibb binlog files included automatically, consistent with data
2. Restore the backup on the replica
   → @@gtid_slave_pos populated from InnoDB state automatically
3. Declare the replica:
   CHANGE MASTER TO
   MASTER_HOST = ‘primary’,
   MASTER_USER = ‘replicator’,
   MASTER_PASSWORD = ‘replpass’,
   MASTER_USE_GTID = slave_pos, ← instruction, not a value
   MASTER_DEMOTE_TO_SLAVE = 1; ← folds local writes into GTID pos
4. START SLAVE;
   → replica announces its GTID set to the primary
   → primary streams from that point forward
   → done

You never touch a file name or a byte offset. The position is embedded in the InnoDB state from the moment the backup was consistent.

Split-Brain Recovery

Split-brain is where this matters most. Consider a scenario where the primary and replica became isolated:

Primary:  committed GTIDs 0-1-1 through 0-1-10042
Replica:  applied GTIDs   0-1-1 through 0-1-10039  (was lagging at split)
          then received 3 rogue writes (briefly promoted to primary)
          now has:  0-1-10040, 0-1-10041, 0-1-10042  (divergent, different transactions)

With FILE binlog, reconciling this required manually scanning both binlog files to find the exact divergence point — line by line, event by event. Many teams simply wiped the replica and re-provisioned from scratch to be safe. Even experienced DBAs would spend hours on this.

With GTID, the divergence is immediately visible and unambiguous:

-- On the drifted replica:
SELECT @@gtid_slave_pos;
-- Returns: 0-1-10042  (but these last 3 GTIDs are rogue, divergent from primary)

-- On the primary:
SELECT @@gtid_binlog_pos;
-- Returns: 0-1-10042  (completely different transactions at 10040-10042)

To resync, declare the last known good position explicitly and restart:

STOP SLAVE;
RESET SLAVE;
SET GLOBAL gtid_slave_pos = '0-1-10039';  -- last confirmed good GTID from primary
START SLAVE;
-- MariaDB streams 10040-10042 from the primary, no manual intervention

With gtid_strict_mode = ON (recommended), MariaDB will refuse to apply a GTID it has already seen with different content — you get an explicit error rather than silent data corruption. The divergence surface is precisely identified, not guessed at.

MASTER_DEMOTE_TO_SLAVE=1 — The Safety Net for Former Primaries

When resetting a server that was briefly acting as a primary (during split-brain, or when repurposing a former primary as a replica), its @@gtid_binlog_pos includes local writes that the current primary does not know about. Without reconciliation, MASTER_USE_GTID = slave_pos would start from @@gtid_slave_pos, which might be behind the local writes — creating a gap.

MASTER_DEMOTE_TO_SLAVE = 1 handles this automatically: it computes the union of @@gtid_slave_pos and @@gtid_binlog_pos, sets that as the new gtid_slave_pos, and proceeds. In plain English: “I may have had local writes — include them in my starting position declaration so I don’t replay things I already have.”

Without MASTER_DEMOTE_TO_SLAVE=1:
— Risk: potential GTID gap or conflict if server had local writes

With MASTER_DEMOTE_TO_SLAVE=1:
— MariaDB automatically: SET gtid_slave_pos = gtid_slave_pos ∪ gtid_binlog_pos
— Then connects to primary from that unified position
— Safe regardless of the server’s previous role

In summary, the GTID + InnoDB binlog combination reduces your split-brain/resync runbook from a multi-step forensic procedure with manual position arithmetic to three commands:

STOP SLAVE;
SET GLOBAL gtid_slave_pos = '<last_known_good_gtid>';
CHANGE MASTER TO MASTER_USE_GTID = slave_pos, MASTER_DEMOTE_TO_SLAVE = 1;
START SLAVE;

For teams (like me) that have been burned by binlog position mismatches in production, this operational simplification was a game changer, and now with adding binlogs into InnoDB WALs, the procedure is position-agnostic. This means a faster and easier way to create a replica and automate a full re-sync because mariadb-backup captures data and binlog atomically and restoring sets @@gtid_slave_pos automatically. No parsing, no conditionals, no positions arithmetic, the same script will work whether you re-sync after lag, crash, or split-brain (which is even less likely) idempotently.

Galera Cluster and InnoDB Cluster: What Changes (and What Doesn’t)

Why Galera Is Blocked and Why It’s Non-Trivial to Fix

Understanding the block requires understanding what Galera actually does at commit time — and crucially, what it does not use the binlog for between cluster nodes.

Galera nodes do not replicate by shipping binlog events to each other. The replication mechanism is the wsrep write set protocol: at commit time, the node extracts the changed rows as a write set, broadcasts it to all cluster nodes, and all nodes run a certification protocol (conflict detection) before any node commits. The commit is synchronous across all nodes.

Galera write path:
  Client writes to Node A
  → InnoDB: prepare transaction
  → wsrep: extract write set from transaction
  → wsrep: broadcast write set to all nodes
  → wsrep: certification round (all nodes vote — conflicts?)
  → wsrep: apply write set on all nodes simultaneously
  → InnoDB: commit on all nodes
  → binlog written AFTER commit (for async slaves attached to the cluster)

The binlog in a Galera cluster has exactly one purpose: feeding async replica slaves that hang off one of the cluster nodes for analytics, backup, or reporting. The Galera nodes themselves never read each other’s binlog.

The problem with InnoDB binlog is the certification hook. The wsrep plugin currently intercepts between InnoDB prepare and InnoDB commit — it sits at the 2PC boundary that InnoDB binlog eliminates. With the new architecture, InnoDB prepare and commit are atomic — there is no pause point where wsrep can insert its certification round. The wsrep plugin needs to be re-integrated at a different layer of the storage engine API, which is a significant engineering effort. The GALERA26 label on MDEV-34705 acknowledges this as planned future work, but it is not present in 12.3.

What wsrep needs:
InnoDB prepare → [wsrep certification] → InnoDB commit

What InnoDB binlog does:
InnoDB prepare + commit = atomic, no pause point→ Incompatible. wsrep cannot insert itself.

Impact on Galera Async Slaves

Even if you are not running Galera nodes with InnoDB binlog, you may be running async replicas attached to a Galera node — a common pattern for offloading analytics queries or running mariadb-backup from a dedicated replica. These are unaffected as long as the Galera node itself continues to use binlog_storage_engine=FILE. The async slave receives standard .bin events from the Galera node and nothing in its pipeline changes.

The restriction is: the Galera node acting as the async replication source must stay on FILE binlog. You cannot mix binlog storage engines within the same replication chain in a meaningful way — the source determines the format.

Production Decision Matrix by Topology

Topology	InnoDB binlog viable?	Notes
Standalone primary + async replicas (GTID)	Yes	Primary use case, fully supported
Primary + async replicas (file/offset)	Not yet	Migrate to GTID first
Semi-sync AFTER_COMMIT + async replicas	Yes	Supported in 12.3.1; AFTER_SYNC is architecturally incompatible
Semi-sync AFTER_SYNC	Never	Architecturally incompatible with 2PC removal
Galera cluster nodes	Not yet	wsrep hook incompatible, planned for future
Async slave off a Galera node	Unaffected	Galera node uses FILE binlog as source
MaxScale read/write split	Test required	Replication protocol unchanged, failover scripts may need GTID update
MySQL InnoDB Cluster	N/A	MySQL-only, not applicable to MariaDB

Known Limitations and Open Issues (12.3.1 RC)

This feature is marked as opt-in for good reason. The following limitations are documented and/or reported in the JIRA tracker. Read these before recommending it for production.

Documented Limitations (by design, 12.3.1)

Open JIRA Issues (as of March 2026)

MDEV-38462 — InnoDB: Crash recovery is broken due to insufficient innodb_log_file_size with new binlog Severity: likely Major. The new binlog writes into the InnoDB redo log, which means large transactions or high write rates can exhaust innodb_log_file_size more rapidly than before. If the redo log is undersized for the binlog load, crash recovery may fail. Lab implication: always set innodb_log_file_size generously (≥2 GB) when enabling the new binlog.

MDEV-38304 — InnoDB Binlog to be Stored in Archived Redo Log A follow-on feature request for archiving binlog data within the InnoDB redo log archiving infrastructure. Not a blocker, but signals that the archival story is incomplete.

MDEV-34705 (parent, closed/fixed in 12.3.1, resolved 2026-02-08) — The main implementation MDEV. Review the sub-tasks and linked issues for outstanding items, notably MDEV-38190 (semi-sync enhancements) and MDEV-38307 (wsrep/Galera feasibility study).

Operational Cautions for Your Lab

• Do not test innodb_flush_log_at_trx_commit=0 on a shared Azure disk — results will be misleading due to OS-level write caching. Use a dedicated Premium SSD P30+ or Ultra Disk.
• The sync_binlog parameter being silently ignored is a foot-gun during migration — add it to your monitoring alerting if teams use it as a proxy for “durable binlog.”
• Pre-allocated 1 GB .ibb files mean disk space consumption “looks” immediately high even under low load.

The Benchmark Design

What We Are Measuring

Three performance dimensions across two configurations:

CONFIG A: binlog_storage_engine=FILE (classic)
CONFIG B: binlog_storage_engine=innodb (new)

Each dimension tested under three durability profiles:

D1: innodb_flush_log_at_trx_commit=1 + sync_binlog=1 (full durability, “gold”)
D2: innodb_flush_log_at_trx_commit=2 + sync_binlog=0 (OS-buffered WAL)
D3: innodb_flush_log_at_trx_commit=0 (no fsync, maximum throughput)

Note: D2/D3 on CONFIG B is where the architectural gain is most visible. D3 on CONFIG A is effectively unsafe for replication; we benchmark it anyway as a theoretical ceiling.

Metrics

Concurrency Levels

• 1 thread (establish single-threaded overhead)
• 8 threads (typical application concurrency)
• 32 threads (high concurrency, group commit regime)
• 64 threads (stress, disk saturation test)

Azure Lab Setup

Host caching was set to None on both P30 data disks to ensure we measure actual I/O latency, not Azure’s host-level read cache. The data directory, InnoDB redo log, and binlog all reside on the P30 data disk (/data/mysql).

Each profile was tested with 2 runs per thread count (averaged). The datadir was fully reinitialized between profiles to prevent cross-contamination.

My Lab Results

The following results were obtained on Azure Standard_E4ds_v5 VMs (4 vCPU, 32 GB RAM) with P30 Premium SSD data disks (500 GB, 5,000 IOPS, host caching disabled). MariaDB 12.3.1 RC installed from binary tarball.

Lab parameters: sysbench oltp_write_only, 4 tables × 1M rows, 60s per run (15s warmup), 2 runs per data point (averaged), 20 GB InnoDB buffer pool, 4 GB redo log.

Write TPS — The D1 Headline: 2.4–3.3× Faster

Even at single-thread concurrency, InnoDB binlog is 2.4× faster. At 64 threads, the gap widens to 3.2× — the elimination of the binlog fsync becomes more valuable as group commit contention increases.

The p99 latency tells the story most clearly: FILE binlog latency rises steeply with concurrency (16 ms → 72 ms), while InnoDB binlog stays dramatically lower (8 ms → 29 ms at 64 threads). The second fsync creates a latency cliff under contention that InnoDB binlog simply does not have.

D2/D3 — Parity When fsyncs Are Already Gone

At D2 (innodb_flush_log_at_trx_commit=2, OS-buffered) and D3 (=0, no fsync), both binlog engines produce nearly identical throughput: ~10,000–11,000 TPS at saturation. The bottleneck shifts from binlog sync overhead to InnoDB page writes and the P30 disk’s 5,000 IOPS ceiling.

This is the control experiment that validates the D1 results: the InnoDB binlog advantage comes specifically from eliminating the sync_binlog=1 fsync. When fsyncs are already absent, there is nothing to eliminate.

Crash Recovery — Same Speed, Different Architecture

All six profiles: zero data loss (1,000,000 rows before crash = 1,000,000 rows after recovery).

Recovery times are comparable (~27–39 seconds). The real difference is architectural:

FILE binlog recovery log shows the classic 2PC reconciliation:

InnoDB: Starting crash recovery from checkpoint LSN=613360854
InnoDB: To recover: 54820 pages
InnoDB: Transaction 1418253 was in the XA prepared state.
InnoDB: Transaction 1418254 was in the XA prepared state.
...

InnoDB binlog recovery log shows single-path recovery — no XA coordination:

InnoDB: Starting crash recovery from checkpoint LSN=6438545394
InnoDB: To recover: 41688 pages
InnoDB: Continuing binlog number 2 from position 583656195.
mariadbd: ready for connections.

The critical operational difference: with FILE binlog at D2/D3 (sync_binlog=0), the binlog can lose events on crash while InnoDB has already committed them — creating silent primary-replica divergence. With InnoDB binlog, this class of failure is architecturally impossible because the binlog is inside the redo log.

Large Transaction Commit — Redo Amplification Is Real

InnoDB binlog writes ~2× the redo log because every binlog event is also an InnoDB redo log record. Despite this, the commit time is actually slightly faster (1.56s vs 1.79s) because there is no separate binlog fsync to wait for.

For OLTP workloads with small transactions, the redo amplification is negligible — the fsync elimination dominates. For bulk ETL operations producing large redo volumes, this overhead is measurable and should be factored into innodb_log_file_size sizing.

Replication Lag — Same Lag, 56% More Throughput

We measured replication lag with a single async replica (E2ds_v5, 2 vCPU) under sustained 16-thread write load for 180 seconds, monitoring Seconds_Behind_Master every 10 seconds.

The lag growth rate is nearly identical despite the primary pushing 56% more transactions with InnoDB binlog. The bottleneck is the replica’s single-threaded SQL apply — it falls behind at approximately the same rate regardless of how fast the primary commits.

elapsed_s | file_d1 lag | innodb_d1 lag
       10 |          6s |           39s
       60 |         58s |           89s
      120 |        121s |          148s
      180 |        185s |          206s
      240 |        249s |          265s
      300 |        313s |          321s

The implication: InnoDB binlog does not make replication lag worse — it makes the primary faster while the replica remains the limiting factor. Enabling slave_parallel_threads on the replica would likely amplify the advantage further, but that is a test for another day.

Analysis and Recommendations

What the Results Mean for Production

The 2.4–3.3× TPS improvement at D1 durability on Azure P30 disks (5,000 IOPS) is not a synthetic artifact — it represents the removal of a real fsync() call from the commit path. On higher-IOPS storage (Azure Ultra Disk at 10K+ IOPS, NVMe at 100K+ IOPS), the absolute TPS numbers will be higher, but the relative improvement should hold because the 2PC fsync overhead is a fixed latency cost per commit group.

The p99 latency curve tells the production story most clearly: FILE binlog p99 climbs from 16 ms to 72 ms as threads increase from 1 to 64, while InnoDB binlog stays at 8 ms to 29 ms. Tail latency stability under increasing concurrency is exactly what application teams need.

The D2/D3 parity results are equally important: they prove the improvement comes specifically from eliminating sync_binlog=1, not from some general overhead reduction. When fsyncs are already absent, InnoDB binlog adds no benefit — but crucially, it adds no penalty either.

When NOT to Use InnoDB Binlog

Despite the compelling performance numbers:

1. Large batch/ETL workloads: The 2× redo amplification on large transactions increases redo log pressure. For workloads dominated by bulk INSERTs or mass UPDATEs, size innodb_log_file_size accordingly.
2. Galera clusters: Not supported in 12.3.1 RC. The wsrep certification hook is incompatible with the new commit path.
3. Semi-sync AFTER_SYNC: Architecturally incompatible — the 2PC boundary that AFTER_SYNC hooks into no longer exists. AFTER_COMMIT is supported and is the only available wait point with InnoDB binlog.
4. Third-party binlog readers: Tools that read .bin files directly will not understand .ibb format. Tools using the replication protocol are unaffected.
5. Disk-constrained environments: Pre-allocated 1 GB .ibb files mean immediate disk usage even under low load.

innodb_log_file_size Recommendation

With binlog_storage_engine=innodb, the redo log absorbs all binlog writes. Our lab used innodb_log_file_size=4G without encountering MDEV-38462. For production with high write throughput, start with 4 GB and monitor SHOW ENGINE INNODB STATUS for checkpoint age approaching the log file size. If crash recovery reports “insufficient innodb_log_file_size”, increase to 8 GB for example and monitor. The 2× redo amplification we measured means your redo log needs to be roughly twice as large as it would be without InnoDB binlog.

Bottom Line

For GTID-based async replication with standard tooling, binlog_storage_engine=innodb delivers a 2.4–3.3× TPS improvement at full durability, with 2–2.5× lower p99 latency under
concurrency. The migration requires GTID (mandatory), gives up file/offset positioning (use MASTER_GTID_WAIT() instead), and needs generous innodb_log_file_size. At relaxed durability
(D2/D3), throughput is identical, but crash consistency is now guaranteed, which was never the case with FILE binlog.

More importantly, this changes the architecture decision matrix. Binlog-based replication has always carried a performance tax, that pushed some teams toward Galera
clusters not because they needed synchronous multi-master or local HA, but simply to avoid that penalty while maintaining an offsite copy for DR. I often meet teams that conflate HA and
DR requirements and end up implementing a 3-node Galera cluster spread across 3 sites — paying the cross-site certification latency on every single commit for the sake of a disaster
recovery copy that doesn’t need to be synchronous. That trade-off no longer exists. With InnoDB binlog, a simple primary + async replica gives you DR capability at effectively the same
write throughput as a standalone server, while keeping your Galera cluster (if you need one) local where certification latency stays low.

For OLTP workloads, this is the single largest performance improvement available in MariaDB 12.3 and it comes with better crash consistency, simpler automation, and a cleaner
separation between HA and DR as a bonus.

References

• MDEV-34705: Improving performance of binary logging by removing the need of syncing it — https://jira.mariadb.org/browse/MDEV-34705
• MDEV-38462: Crash recovery broken with insufficient innodb_log_file_size (new binlog) — https://jira.mariadb.org/browse/MDEV-38462
• MariaDB 12.3 Changes & Improvements — https://mariadb.com/docs/release-notes/community-server/12.3/mariadb-12.3-changes-and-improvements
• Official binlog implementation documentation (knielsen branch) — https://github.com/MariaDB/server/blob/knielsen_binlog_in_engine/Docs/replication/binlog.md
• Mark Callaghan’s benchmark of the new binlog feature — referenced in the MariaDB.org release blog
• MariaDB Foundation: New binlog implementation blog post — https://mariadb.org/tag/mariadb-releases/

L’article MariaDB 12.3 – Binlog Inside InnoDB est apparu en premier sur dbi Blog.

Step 1: Installation

First the servers will be prepared for the installation. This includes different packages to be installed and ports to be opened.

galera-cluster-1a:~$ sudo apt update
galera-cluster-1a:~$ sudo apt-get install ufw
galera-cluster-1a:~$ sudo ufw enable
galera-cluster-1a:~$ sudo ufw allow 3306
galera-cluster-1a:~$ sudo ufw allow 4444
galera-cluster-1a:~$ sudo ufw allow 4567
galera-cluster-1a:~$ sudo ufw allow 4568
galera-cluster-1a:~$ sudo ufw allow 22

Installing MariaDB 11.3.2 and other packages

galera-cluster-1a:~$ sudo apt-get install libsnmp-perl -y
galera-cluster-1a:~$ sudo curl -LsS -O https://downloads.mariadb.com/MariaDB/mariadb_repo_setup
galera-cluster-1a:~$ sudo bash mariadb_repo_setup --os-type=debian --os-version=bullseye --mariadb-server-version=11.3.2
galera-cluster-1a:~$ sudo wget http://ftp.us.debian.org/debian/pool/main/r/readline5/libreadline5_5.2+dfsg-3+b13_amd64.deb
galera-cluster-1a:~$ sudo dpkg -i libreadline5_5.2+dfsg-3+b13_amd64.deb
galera-cluster-1a:~$ sudo apt-get update
galera-cluster-1a:~$ sudo apt-get install mariadb-server mariadb-client -y
galera-cluster-1a:~$ sudo systemctl enable mariadb
sudo apt-get install percona-toolkit -y

Now you can login to MariaDB and check the version

galera-cluster-1a:~$ mariadb
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 32
Server version: 11.3.2-MariaDB-1:11.3.2+maria~deb11 mariadb.org binary distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

For the final touch for the basic installation of MariaDB we are going to secure it. Answering yes to all the changes it proposes.

galera-cluster-1a:~$ sudo mariadb-secure-installation 

Step 2: Configuring MariaDB server

We recommend creating a dedicated backup user for mariabackup like this:

galera-cluster-1a:~$ mariadb
MariaDB [(none)]> create user 'mariabackup'@'%' identified by 'XXX';
Query OK, 0 rows affected (0.001 sec)
MariaDB [(none)]> GRANT RELOAD, PROCESS, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'mariabackup'@'%';
Query OK, 0 rows affected (0.001 sec)
MariaDB [(none)]> create user 'mariabackup'@'localhost' identified by 'XXX';
Query OK, 0 rows affected (0.001 sec)
MariaDB [(none)]> GRANT RELOAD, PROCESS, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'mariabackup'@'localhost';
Query OK, 0 rows affected (0.001 sec)
MariaDB [(none)]> flush privileges;
MariaDB [(none)]> exit

Creating a dedicated directory for logs.

sudo mkdir /var/log/mariadb
sudo chown mysql:mysql /var/log/mariadb 

In the /etc/mysql/mariadb.conf.d/50-server.cnf file we add the following options at the end to the [mariadb-11.3] section.

# This group is only read by MariaDB-11.3 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand
[mariadb-11.3]
log_warnings=2
log_error=/var/log/mysql/mariadb.err
general_log = 1
general_log_file = /var/log/mysql/mariadb.log

[mysqldump]
user=dump
password=XXX
max_allowed_packet = 512M

[mariabackup]
user=mariabackup
password=XXX
databases-exclude=lost+found

If you want to use mariabackup for backups and replication you will have to create the user and add the last section [mariabackup]

Step 3: Configuring Galera Cluster

Change the galera config file to the following. It is also possible to use the IPs instead of hostnames.

galera-cluster-1a:~$ cat /etc/mysql/mariadb.conf.d/60-galera.cnf
[galera]
# Mandatory settings
wsrep_on = ON
wsrep_cluster_name = MariaDB Galera Cluster
wsrep_cluster_address = gcomm://galera-cluster-1a,galera-cluster-1b,galera-cluster-1c
wsrep_provider = /usr/lib/galera/libgalera_smm.so
binlog_format = row
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2
# Allow server to accept connections on all interfaces.
bind-address = 0.0.0.0

# Optional settings
wsrep_slave_threads = 1
innodb_flush_log_at_trx_commit = 0
wsrep_sst_auth = mariabackup:manager
wsrep_sst_method=mariabackup
wsrep_node_address=galera-cluster-1a
wsrep_sst_receive_address= galera-cluster-1a

Step 4: Bootstrapping Galera Cluster

Now everything is ready and we can start the MariaDB Server in cluster mode.

galera-cluster-1a:~$  sudo systemctl stop mariadb.service
galera-cluster-1a:~$  sudo galera_new_cluster

Lets check if the one node cluster has successfully started. The start position on the very first start should look like the following

galera-cluster-1a:~$ ps -f -u mysql
UID        PID  PPID  C STIME TTY          TIME CMD
mysql     2081     1  0 14:37 ?        00:00:00 /usr/sbin/mariadbd --wsrep-new-cluster --wsrep_start_position= 00000000-0000-0000-0000-000000000000:-1 

Login to your instance and check the cluster size

galera-cluster-1a:~$ mariadb

MariaDB [(none)]> show global status like 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 1     |
+--------------------+-------+
MariaDB [(none)]> quit

Step 5: Setting up the other servers

Repeat steps 1-3 on the other two nodes but you do not have to bootstrap these nodes.
Simply restart the mariadb.service at the end for them to sync with the cluster.

galera-cluster-1b:~$ sudo systemctl restart mariadb.service
galera-cluster-1c:~$ sudo systemctl restart mariadb.service 

Afterwards you can check the cluster size again.

galera-cluster-1a:~$ mariadb
MariaDB [(none)]> show global status like 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 3     |
+--------------------+-------+

Check out this blog about using mariabackup to repair your replication: https://www.dbi-services.com/blog/mariadb-repair-replication-through-db-restore-mariabackup/

L’article Setting up MariaDB Galera Cluster est apparu en premier sur dbi Blog.

After a welcome coffee to meet each other, Pascal Zanette shortly introduced the event and the agenda of the day.

Our CTO Hervé Schweitzer presented then YaK, developed by dbi services.
The idea was born 2 years ago, to find a solution to deploy Oracle instances on several Cloud providers (Oracle, Microsoft Azure, AWS) and On-Premises:
– in the same heterogeneous way
– avoiding constraints of each Cloud providers solution
– automated through Ansible.
YaK is made of 2 parts:
– YaK Core – as an Open Source collection (for the central inventory and instances deployment)
– YaK Components – based on a subscription (for database deployment and with dbi services best practices integration).
Hervé showed how YaK works more in details through a demo, that you can play again here.
YaK is already available for Oracle and PostgreSQL. YaK for SQL Server, MariaDB, Kubernetes, WebLogic, MongoDB and Alfresco is under development and will come soon, so stay tuned!

Marc Wagner presented how to migrate from On-Premises to Oracle OCI using Zero Downtime Migration tool (ZDM).
Marc talked about advantages to go to the Cloud (such as scalability and flexibility, no dedicated hardware, less responsibility), but also alerted the audience about things to think about (such as performances and latency, costs monitoring, lock-in risk, sensitive data). Once we decided to make this change, what about downtime to migrate to the Cloud? That’s why ZDM is interesting: it’s high available (applying MAA Best Practices), it offers a very short downtime and a minimal impact on production systems and reduces human errors. ZDM allows to migrate to Oracle OCI but also to On-Premises Exadata, Exadata Cloud@Customer (ExaCC), Autonomous Database, etc…
And finally Marc presented through a live demo how to use ZDM with an Offline Logical Migration workflow to migrate from On-Premises to ADB (you can find a summary here).

As a last session of the event, Marc Keller, with Frank Zeindler remotely, presented Cloud migration made easier and safer with data validation. Marc explained why an extra validation after a data migration is helpful and how the OMrun framework can compare data and orchestrate various processes for this purpose. Some OMrun use cases were then explored:
– Quality assurance for reporting
– Data integrity monitoring for system changeover and production
– Data crawling and data cleansing
– Data masking
– Data quality assurance during migration to the Cloud.

The event is over. After a nice lunch break with colleagues, customers and attendees, we can come back home keeping in mind the magic triple jump to go to the Cloud (as Marc mentioned during his session):
– dbi services YaK
– Oracle ZDM
– OMrun.

Last thing: don’t forget that another “Oracle Cloud Infrastructure: everything for an efficient migration” event will take place on May 23 in Basel.
And some other dbi services events will be organized during the year about DevOps, Cloud, Data and much more. So again, stay tuned!

By Elisa Usai

L’article An overview of “Oracle Cloud Infrastructure: everything for an efficient migration” event by dbi services est apparu en premier sur dbi Blog.

In my example, let’s consider that I’ve a server01 and server02, both running on Ubuntu 20.04.

server02 is currently the master, server01 the slave.

We’ll perform a switchover to come back to a normal status, meaning server01 as master and server02 as slave.

Stop all the connections to the DB

root@server02:~# mysql
mysql> flush tables with read lock;

Recovering information from the master

Note the values for the parameters File and Position.

root@server02:~# mysql
mysql> SHOW MASTER STATUS\G;

***** 1. row *****
File: mysql-bin.007747
Position: 5512879
Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set:
1 row in set (0.00 sec)

Check slave status

We check that the replication is working properly and that the values for the parameters Master_Log_File and Read_Master_Log_Pos correspond to the values noted in point 1.3.

root@server01:~# mysql
mysql> SHOW SLAVE STATUS\G;

Stop slave and do the Switchover

From this step, the slave should switch back to master.

Stop the slave, reset the slave configuration to 0, reset the master configuration to 0 and switch over.

Finally we check that the changes have been made.

root@server01:~# mysql
mysql> STOP SLAVE;
mysql> reset slave all;
mysql> RESET MASTER;
mysql> SHOW SLAVE STATUS\G;
mysql> SHOW MASTER STATUS\G

Change configuration in my.cnf

In the configuration of the instance, we set the slave as read only and the opposite for the master.

### On Master ###
root@server01:~# cd /etc/mysql/
root@server01:/etc/mysql# vi mysql.conf.d/mysqld.cnf

# In my.cnf:
read_only=0
super_read_only=0

### On slave ###
root@server02:~# cd /etc/mysql/
root@server02:/etc/mysql# vi mysql.conf.d/mysqld.cnf

# In my.cnf:
read_only=1
super_read_only=1

Restart mysql service

Restart the service to apply the changes in my.cnf

### On Master ###
# Stop the MySQL service
root@server01:~# systemctl stop mysql.service

# Start the MySQL service
root@server01:~# systemctl start mysql.service

### On slave ###
# Stop the MySQL service
root@server02:~# systemctl stop mysql.service

# Start the MySQL service
root@server02:~# systemctl start mysql.service

Configure replication on new slave

root@server02:~# mysql
mysql> CHANGE MASTER TO MASTER_HOST='MASTER IP', MASTER_USER='repl', MASTER_PASSWORD='***';
mysql> START SLAVE;

VIP address swap

If the VIP address is on the server with the slave instance, we set up the VIP on the server with the master instance.

### On Slave ###
root@server02:~# ip addr del VIP.AD.DR.ESS/28 dev ens160

### On Master ###
root@server01:~# ip addr add VIP.AD.DR.ESS/28 dev ens160

### If blocked by firewall ###
iptables -I INPUT -i ens160 -d VIP.AD.DR.ESS -p icmp --icmp-type echo-request -j ACCEPT

Conclusion

The cluster is now back to a normal status, with server01 as master and server02 as slave.

L’article MariaDB manual Switchover est apparu en premier sur dbi Blog.

A tale of 2 Oracle drivers

Oracle’s drivers support 2 implementations: the pure java one as used above, also named Thin drivers, with the following URL syntax:

jdbc:oracle:thin:@//db_machine:listener_port/service_name

but also the so-called OCI-based or Thick Type 2 drivers that rely on SQL*Net and the connect strings defined in tnsnames.ora, with the following syntax:

jdbc:oracle:oci:@connect_string

Since they use OCI native libraries, they are said to be faster than their pure java implementation counterpart. To have those libraries, an InstantClient is needed at the minimum. Here, I’m using an existing InstantClient v21.7 for Linux installation /u01/dctm/repo01/oracle/instantclient_21_7. It comes with its own jdbc drivers which are incompatible with the ones downloaded before (but can do both Thin and OCI), so let’s switch to the bundled ones. As they depend on native OCI libraries, those must be made accessible through the LD_LIBRARY_PATH.

Here is the definition of the connect string mypdb1 we will use; it has the same settings as the ones explicitly used with the Thin drivers:

mypdb1 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.0.21)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = pdb1)
    )
  )

Let’s adapt the environment and syntax, and try:

$ export LD_LIBRARY_PATH=/u01/dctm/repo01/oracle/instantclient_21_7
$ java -cp /u01/dctm/repo01/oracle/instantclient_21_7/ojdbc8.jar:. jdbc_tester_generic oracle.jdbc.driver.OracleDriver repo01 repo01 jdbc:oracle:oci:@mypdb1 "select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual"
The command-line parameters are:
arg 0 = oracle.jdbc.driver.OracleDriver
arg 1 = repo01
arg 2 = repo01
arg 3 = jdbc:oracle:oci:@mypdb1
arg 4 = select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual

Loading driver [oracle.jdbc.driver.OracleDriver]:
JDBC Driver oracle.jdbc.driver.OracleDriver successfully loaded

Connecting to url [jdbc:oracle:oci:@mypdb1] as user [repo01]:
Successful connection to database

Executing SQL query [select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual]:

Retrieving the result set:
now in Oracle db: 22/01/2023 15:51:23

JDBC connection successfully tested, quitting …

As illustrated, since those drivers are based on OCI, they go through the SQL*Net stack to read the connectivity configuration behind the connect string mypdb1, namely from the tnsnames.ora file; hence this information does not need to be provided any more, which reflects in the simpler JDBC URL. Whenever the Oracle client is installed along with JDBC programs, it is preferable to use the OCI drivers to avoid replicating connection information instead of accessing it from one single place, although when working with RDBMS from different vendors, it is understandable to standardize to the Thin drivers for generality and portability. Anyway, there are sufficient differences in the URL syntax across RDBMS that one more does not change much. For example, Oracle Thin drivers also allows to specify the URL through the full, inlined tnsnames.ora syntax, e.g.:

$ export LD_LIBRARY_PATH=/u01/dctm/repo01/oracle/instantclient_21_7
$ java -cp /u01/dctm/repo01/oracle/instantclient_21_7/ojdbc8.jar:. jdbc_tester_generic oracle.jdbc.driver.OracleDriver repo01 repo01 jdbc:oracle:thin:@"(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP) (HOST=192.168.0.21)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=pdb1)))" "select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual"
The command-line parameters are:
arg 0 = oracle.jdbc.driver.OracleDriver
arg 1 = repo01
arg 2 = repo01
arg 3 = jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP) (HOST=192.168.0.21)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=pdb1)))
arg 4 = select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual

Loading driver [oracle.jdbc.driver.OracleDriver]:
JDBC Driver oracle.jdbc.driver.OracleDriver successfully loaded

Connecting to url [jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP) (HOST=192.168.0.21)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=pdb1)))] as user [repo01]:
Successful connection to database
...
JDBC connection successfully tested, quitting …

and so do the OCI drivers:

$ export LD_LIBRARY_PATH=/u01/dctm/repo01/oracle/instantclient_21_7
$ java -cp /u01/dctm/repo01/oracle/instantclient_21_7/ojdbc8.jar:. jdbc_tester_generic oracle.jdbc.driver.OracleDriver repo01 repo01 jdbc:oracle:oci:@"(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP) (HOST=192.168.0.21)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=pdb1)))" "select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual"
The command-line parameters are:
arg 0 = oracle.jdbc.driver.OracleDriver
arg 1 = repo01
arg 2 = repo01
arg 3 = jdbc:oracle:oci:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP) (HOST=192.168.0.21)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=pdb1)))
arg 4 = select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual

Loading driver [oracle.jdbc.driver.OracleDriver]:
JDBC Driver oracle.jdbc.driver.OracleDriver successfully loaded

Connecting to url [jdbc:oracle:oci:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP) (HOST=192.168.0.21)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=pdb1)))] as user [repo01]:
Successful connection to database
...
JDBC connection successfully tested, quitting ...

Using a SID with Oracle JDBC

Oracle JDBC drivers allow to specify a service name, as shown so far, but also the deprecated, old-style SID. The URL syntax changes slighty. Here is a summary of the all the Oracle JDBC URL syntaxes:

# with a service name;
# Thin:
jdbc:oracle:thin:@//192.168.0.21:1521/pdb1
# OCI:
jdbc:oracle:oci:@mypdb1

# with a sid;
# Thin:
jdbc:oracle:thin:@192.168.0.21:1521:orcl
# OCI:
jdbc:oracle:oci:@mysid

with the following mysid‘s definition in tnsnames.ora:

mysid =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = db)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SID = orcl)
    )
  )

where orcl is the sid of the container database. As we are accessing the container database now, and the repo01 account does not exist there, we must switch to a different account; let’s use system as it is defined everywhere (however, it is still possible to access a pluggable database through its SID, and use the repo01 account; for this, set the listener’s parameter USE_SID_AS_SERVICE_listener=on; I did not try it though and I suspect that the service syntax will have to be used but with a SID instead of a service name).

Let’s adapt the environment and syntax, and try first with the OCI drivers:

$ export LD_LIBRARY_PATH=/u01/dctm/repo01/oracle/instantclient_21_7
$ java -cp /u01/dctm/repo01/oracle/instantclient_21_7/ojdbc8.jar:. jdbc_tester_generic oracle.jdbc.driver.OracleDriver system manager jdbc:oracle:oci:@mysid "select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual"
The command-line parameters are:
arg 0 = oracle.jdbc.driver.OracleDriver
arg 1 = system
arg 2 = manager
arg 3 = jdbc:oracle:oci:mysid
arg 4 = select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual

Loading driver [oracle.jdbc.driver.OracleDriver]:
JDBC Driver oracle.jdbc.driver.OracleDriver successfully loaded

Connecting to url [jdbc:oracle:oci:@mysid] as user [system]:
Successful connection to database
...
JDBC connection successfully tested, quitting ...

Using the full inline definition behind the mysid connect string:

$ export LD_LIBRARY_PATH=/u01/dctm/repo01/oracle/instantclient_21_7
$ java -cp /u01/dctm/repo01/oracle/instantclient_21_7/ojdbc8.jar:. jdbc_tester_generic oracle.jdbc.driver.OracleDriver system manager jdbc:oracle:oci:@"(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP) (HOST=192.168.0.21)(PORT=1521))(CONNECT_DATA=(SID = orcl)))" "select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual"
The command-line parameters are:
arg 0 = oracle.jdbc.driver.OracleDriver
arg 1 = system
arg 2 = manager
arg 3 = jdbc:oracle:oci:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP) (HOST=192.168.0.21)(PORT=1521))(CONNECT_DATA=(SID = orcl)))
arg 4 = select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual

Loading driver [oracle.jdbc.driver.OracleDriver]:
JDBC Driver oracle.jdbc.driver.OracleDriver successfully loaded

Connecting to url [jdbc:oracle:oci:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP) (HOST=192.168.0.21)(PORT=1521))(CONNECT_DATA=(SID = orcl)))] as user [system]:
Successful connection to database
...
JDBC connection successfully tested, quitting ...

Let’s try now this SID with the Thin driver:

$ java -cp ojdbc11.jar:. jdbc_tester_generic oracle.jdbc.driver.OracleDriver system manager jdbc:oracle:thin:@192.168.0.21:1521:orcl "select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual"
The command-line parameters are:
arg 0 = oracle.jdbc.driver.OracleDriver
arg 1 = system
arg 2 = manager
arg 3 = jdbc:oracle:thin:@192.168.0.21:1521:orcl
arg 4 = select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual

Loading driver [oracle.jdbc.driver.OracleDriver]:
JDBC Driver oracle.jdbc.driver.OracleDriver successfully loaded

Connecting to url [jdbc:oracle:thin:@192.168.0.21:1521:orcl] as user [system]:
Successful connection to database
...
JDBC connection successfully tested, quitting ...

Using the inline connect string definition with the Thin drivers:

$ java -cp ojdbc11.jar:.:. jdbc_tester_generic oracle.jdbc.driver.OracleDriver system manager jdbc:oracle:thin:@"(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP) (HOST=192.168.0.21)(PORT=1521))(CONNECT_DATA=(SID = orcl)))" "select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual" 
The command-line parameters are:
arg 0 = oracle.jdbc.driver.OracleDriver
arg 1 = system
arg 2 = manager
arg 3 = jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP) (HOST=192.168.0.21)(PORT=1521))(CONNECT_DATA=(SID = orcl)))
arg 4 = select 'now in Oracle db: ' || to_char(sysdate, 'DD/MM/YYYY HH24:MI:SS') as now from dual

Loading driver [oracle.jdbc.driver.OracleDriver]:
JDBC Driver oracle.jdbc.driver.OracleDriver successfully loaded

Connecting to url [jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP) (HOST=192.168.0.21)(PORT=1521))(CONNECT_DATA=(SID = orcl)))] as user [system]:
Successful connection to database
...
JDBC connection successfully tested, quitting ...

As showed above, the utility jdbc_tester_generic is URL- and drivers-agnostic; just provide the right parameters and they will be passed unchanged to the drivers.

Other RDBMS

We tested jdbc_tester_generic with PostgreSQL and Oracle and chances are that it is really generic indeed as it could handle Oracle’s multiple specificities. Nonetheless, let’s try it with 7 more RDBMS, e.g. MariaDB/MySQL, SQLite, Firebird, Microsoft SQL Server, HSQLDB, MongoDB and Excel, and see how it behaves. If you don’t have those installed somewhere, concise instructions to do it are presented in several articles published on the dbi-services blog site https://www.dbi-services.com/blog/installing-the-odbc-driver-manager-with-*, e.g. https://www.dbi-services.com/blog/installing-the-odbc-driver-manager-with-sqlite-on-linux/ for SQLite.

After the installation of the software and the creation of the databases, their readiness can be verified via ODBC as explained in the articles. Hereafter, we assume that those databases are up and running, with valid credentials where applicable, and we will try to access them via JDBC this time.

MariaDB

First, let’s install the JDBC drivers for MariaDB:

$ wget https://dbschema.com/jdbc-drivers/MariaDbJdbcDriver.zip
$ unzip MariaDbJdbcDriver.zip

The server’s default port is 3306 but let’s check it:

$ netstat -tlnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
...
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 112172/mariadbd

# or:
$ grep -i 'port =' /etc/mysql/my.cnf
$ port = 3306

# or
$ mysql
show variables like 'port';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| port          | 3306  |
+---------------+-------+
1 row in set (0.006 sec)

Let’s invoke the tester now:

$ java -cp mariadb-java-client-3.0.6.jar:. jdbc_tester_generic org.mariadb.jdbc.Driver debian debian jdbc:mariadb://localhost:3306/sampledb "select concat('now in mysql db: ', DATE_FORMAT(SYSDATE(), '%d/%m/%Y %H:%i:%s')) as now;"
The command-line parameters are:
arg 0 = org.mariadb.jdbc.Driver
arg 1 = debian
arg 2 = debian
arg 3 = jdbc:mariadb://localhost:3306/sampledb
arg 4 = select concat('now in mysql db: ', DATE_FORMAT(SYSDATE(), '%d/%m/%Y %H:%i:%s')) as now;

Loading driver [org.mariadb.jdbc.Driver]:
JDBC Driver org.mariadb.jdbc.Driver successfully loaded

Connecting to url [jdbc:mariadb://localhost:3306/sampledb] as user [debian]:
Successful connection to database

Executing SQL query [select concat('now in mysql db: ', DATE_FORMAT(SYSDATE(), '%d/%m/%Y %H:%i:%s')) as now;]:

Retrieving the result set:
now in mysql db: 23/01/2023 21:13:00

JDBC connection successfully tested, quitting ...

No surprise with MariaDB.

SQLite

First, let’s install the JDBC drivers for SQLite:

$ wget https://repo1.maven.org/maven2/org/xerial/sqlite-jdbc/3.40.0.0/sqlite-jdbc-3.40.0.0.jar

Let’s invoke the tester:

$ java -cp sqlite-jdbc-3.40.0.0.jar:. jdbc_tester_generic java.sql.Driver debian debian jdbc:sqlite:/path/to/database/sampledb "select 'now in sqlite db: ' || STRFTIME('%d/%m/%Y, %H:%M:%S', datetime()) as now;"
The command-line parameters are:
arg 0 = java.sql.Driver
arg 1 = debian
arg 2 = debian
arg 3 = jdbc:sqlite:/path/to/database/sampledb
arg 4 = select 'now in sqlite db: ' || STRFTIME('%d/%m/%Y, %H:%M:%S', datetime()) as now;

Loading driver [java.sql.Driver]:
JDBC Driver java.sql.Driver successfully loaded

Connecting to url [jdbc:sqlite:/path/to/database/sampledb] as user [debian]:
Successful connection to database

Executing SQL query [select 'now in sqlite db: ' || STRFTIME('%d/%m/%Y, %H:%M:%S', datetime()) as now;]:

Retrieving the result set:
now in sqlite db: 23/01/2023, 22:02:13

JDBC connection successfully tested, quitting ...

No surprise with SQLite either.

See Part III.

L’article A generic jdbc tester (part II) est apparu en premier sur dbi Blog.

It’s Open Source!

Even if money is not everything…

PMM is completely Open Source, and this is not surprising coming from Percona. But as we know, the advantage of Open Source is not only the fact of being free…

Let’s contribute

One of the keys of Open Source is contribution. When I started using PMM and sometimes I cannot find all the information I needed and more details in the official documentation, or when I discover a bug or have a question, or even if I would like to have a new feature implemented, a whole community is there to support and share knowledge in a very impressive short time:
– A JIRA issue tracker to submit a bug report.
– A very useful blog with plenty of posts about new features, technical step-by-step deployment information, tips and much more. And obviously the possibility to write to the author.
– A Percona Community Forum to exchange with Percona experts.

Evolution

Open Source means also quick evolution of the product. PMM is getting better every month, with continuous improvements, new features, corrections of bugs and so on.

One tool to monitor several technologies

If we speak about monitoring solutions, what makes the difference nowadays? Except rare cases, each of us works with a mix of different database solutions: Open Source vs proprietary, On-Premises vs Cloud. And when we want to identify performance issues on our application, we also need system benchmarks, replication metrics, etc… PMM is a platform that we can use to centralize our monitoring data, because:
– Exporters to collect metrics for MySQL, MariaDB, PostgreSQL and MongoDB instances and their replication, the ProxySQL and the HAProxy, instances hosted on Amazon RDS or on Google Cloud Platform are embedded in the PMM tool as well as pre-configured dashboards, so the monitoring can be easily put in place.
– Even proprietary technologies such as Oracle and SQL Server can be integrated to PMM through external exporters. In this case we have the choice to develop our own exporter and design our own dashboards or use what is already been developed and designed by other contributors. Here you can find the links to some Third-party exporters.

Less maintenance tasks

The PMM platform is based on a client-server model. The 2 main components on the server side are the following ones:
– VictoriaMetrics, a time series database (replacing Prometheus since PMM 2.12.0 version) that aggregates metrics collected by exporters
– Grafana allowing to visualize the aggregated data in a web interface.
But actually, we don’t have to care about that! We don’t need to install Prometheus or VictoriaMetrics components, install Grafana and then configure it to let them talk together. We can just see the PMM platform as a black box: we will have only one tool to install and to maintain.

Sexy dashboards

Integrated dashboards are designed with accuracy and provide a detailed and temporal analysis of our data.
Here my favorite ones:
– Node Summary
It’s the system dashboard. We can visualize here metrics such as CPU, memory, disk occupation and performances, processes, network traffic and get some information about our system architecture:

We have also the possibility to compare several nodes:

– MySQL Summary
It’s one of the MySQL dashboards which displays general metrics about our instance (uptime, version, InnoDB Buffer Pool size, connections, threads, table locks, traffic, and much more):

And we still have the option to compare different instances.
– MySQL Replication
It’s the dashboard which offers us an overview on the MySQL Master-Slave Replication:

– Query Analytics
QAN is a special dashboard that allows us to analyze MySQL and PostgreSQL databases queries over time:

This can be very useful to identify performance problems.

Coming soon

Currently the last PMM stable release is 2.25.0. As we told just above, the tool is evolving a lot, and nice features are already available in technical preview:

You can already play with that even if for the moment it’s not suggested to use them for production environments. I tested for example the Integrated Alerting and I cannot wait to have it available as a stable feature (we don’t need then anymore to configure an External Prometheus Alertmanager to be alerted if something goes wrong on our systems).

I hope you have enjoyed this introduction to PMM.
Now it’s up to you: don’t hesitate to test it! For my part, I will surely write again about PMM, so stay tuned!

L’article What I really like about Percona PMM est apparu en premier sur dbi Blog.

A l’heure du serverless, le futur va-t-il aller aux bases de données distribuées?

Franck Pachot makes a comparison between Oracle products (Active Data Guard, RAC, Sharding) and new distributed databases in order to scale-up and scale-out.
Briefly his talk makes reference to:
– Differences among RDBMS, NoSQL and NewSQL according to the CAP Theorem
– Definition and needs for NoSQL and NewSQL
– Definition of services such as Google’s Cloud Spanner, TiDB, CockroachDB, YugabyteDB.

From DBA to Data Engineer – How to survive a career transition?

Kamran Agayev from Azerbaijan speaks about what Big Data in general is and the transition from DBA to Database Engineer.
He addresses several interesting topics:
– Definition of Big Data
– Which are the skills for a Data Engineer, a Data Architect (more complex competences than “just” being a Database Administrator)
– Definition of products like Hadoop, Kafka, NoSQL
After the coffee break, the choice is between 2 different streams. Here some words about the sessions I attend.

Amazing SQL

Laetitia Avrot from EnterpriseDB talks about SQL, which is much more than what we know. SQL is different from other programming languages but it must be treated as one of them. At school we still learn SQL from before 1992, but in 1999 it changed to add relational algebra and data atomicity. PostgreSQL is very close to this standard. Laetitia shows lots of concrete examples of subqueries, common table expressions (CTE), lateral joins (not implemented in MySQL for the moment), anti joins, rollup, window functions, recursive CTE, and also some explanations about key words such as in, values, not in, not exists.

Graph Database

After the lunch, Gianni Ceresa presents property graph databases as combination of vertex (node, properties, ID) and edge (node, ID, label, properties). To start working with Oracle graphs, we can use PGX (Oracle Labs Parallel Graph AnalytiX). The OTN version is better for documentation. Through a demo, Gianni shows how to build a graph using Apache Zeppelin as interpreter and Phyton and Jupyter to visualize it. And then we can also use it to write some data analysis.

5 mistakes that you’re making while presenting your data

Luiza Nowak, a non-IT girl working with IT people (she is a board member of POUG), talks about IT presentations. There are 4 important parts defining them: the content, the story, the speaker performance and visualization.
Here recurrent errors of IT presentations and how to handle them:
1. Lack of data selection – you need to filter your data, to consider who and where you are talking to
2. Too much at once – you need to divide your content, create some suspense and put less information into slides to let the audience listen to you instead of reading them
3. Forget about contrast – you have to use contrast on purpose because it can be useful to underline something but it can also distract your audience
4. Wrong type of charts – you have to be clear about your data and explain results
5. You don’t have any story – you need to conceptualize your data.

How can Docker help a MariaDB cluster for disaster/recovery

Saïd Mendi from dbi services explains what a MariaDB Galera Cluster is and his benefits and how Docker can help in some critical situations. Actually you can create some delayed slaves which can be useful to emulate the flashback functionality.

Conclusion

SOUG Day arrives to an end. It was a nice opportunity to meet international speakers, discuss with some customers and colleagues, learn and share. As usual, this is part of dbi services spirit and matches with our values!
And now, I have to say you goodbye: it’s the aperitif and dinner time with the community Hope to see you at the next SOUG event.

L’article Some words about SOUG Day in Lausanne est apparu en premier sur dbi Blog.

Previously, everything worked fine with 10.3.14. So, what happened to our system?

Lets’ quickly start an instance:

mysql@vmoel:/u01/app/mysql/product/mariadb-10.3.18/bin/ [mysqld3] ./mysqld_multi start 3
elseif should be elsif at /u01/app/mysql/product/mariadb-10.3.18/bin/mysqld_multi line 352.
syntax error at /u01/app/mysql/product/mariadb-10.3.18/bin/mysqld_multi line 353, near ")
      {"
syntax error at /u01/app/mysql/product/mariadb-10.3.18/bin/mysqld_multi line 356, near "else"
syntax error at /u01/app/mysql/product/mariadb-10.3.18/bin/mysqld_multi line 404, near "}"
Illegal declaration of subroutine main::stop_mysqlds at /u01/app/mysql/product/mariadb-10.3.18/bin/mysqld_multi line 416.

Actually, the error message is quite simple to understand and to fix. We let you check & fix the bug by yourself (hint : “elseif should be elsif”)

That’s it for now but does your MariaDB instances start now? Hum, try it and check your error log like we did:

2019-10-30 20:53:33 0 [Note] InnoDB: 10.3.18 started; log sequence number 1630952; transaction id 21
2019-10-30 20:53:33 0 [Note] InnoDB: Loading buffer pool(s) from /u02/mysqldata/mysqld3/ib_buffer_pool
2019-10-30 20:53:33 0 [Note] Plugin 'FEEDBACK' is disabled.
2019-10-30 20:53:33 0 [ERROR] /u01/app/mysql/product/mariadb-10.3.18/bin/mysqld: unknown variable 'defaults-group-suffix=mysqld3'
2019-10-30 20:53:33 0 [ERROR] Aborting

Still not working ? Hum, now after some further analysis of the perl code especially the perl routine “start_mysqlds” we could identify the real problem.

While, writing this article we found someone already reported that bug MDEV-20728 !  Thanks to him

The fix is directly available on the MariaDB GitHub repository

Good luck

L’article MariaDB 10.3.1x + 10.4.x : mysqld_multi not working properly est apparu en premier sur dbi Blog.