The DMK is a set of standardized tools that aims at easing the work of DBAs by having dbi’s best practices embedded in common scripts across all the database servers of an organization.

dbi services provides the DMK (Database Management Kit) to its customers for multiple technologies: Oracle, Postgres, MongoDB, etc. This toolkit is provided free of charge to all clients who work with dbi services on a consulting project.

New features of the MongoDB DMK in 2.5.0

Sharding support

The biggest addition in DMK 2.5.0 is full sharding support. DMK can now create, start, stop, restart, and display the status of every component of a sharded MongoDB cluster: config server replica sets, shard replica sets, and mongos routers.

Internally, instance management was refactored into a class hierarchy: a MongoInstance base class handles what is common to all MongoDB processes (URI construction, mongosh command execution, status gathering), and MongodInstance / MongosInstance extend it for their respective types. From a day-to-day usage perspective, this is transparent: you use the same dmk_db_ctl.py and dmk_status.py commands for all instance types.

A few behaviours are specific to mongos: stopping a mongos router uses SIGTERM instead of the --shutdown command (which mongos does not support), and trying to back up a mongos instance now raises an explicit error, since mongos processes hold no data. In a sharded configuration, the underlying shards should be backed up instead.

New configuration templates for each sharding component type are available in $DMK_HOME/templates/dbcreate/, in both forked (no systemd) and non-forked (systemd-managed) variants. A dedicated systemd unit file template (mongos.service.template) was also added for mongos routers.

Three new generic JavaScript scripts in $DMK_HOME/js/ handle the initialization steps:

• csrs_initiate.js — initiate a config server replica set
• rs_initiate.js (updated) — initiate a shard or standalone replica set, now with optional arbiter support
• sh_addshards.js — register shards in the cluster via a mongos router

All three receive their parameters via the mongosh --eval flag, keeping them reusable across any cluster topology:

# Initiate the config server replica set
mongosh 127.0.0.1:27017 \
  --eval "var rsName='csrs'; var rsHosts=['127.0.0.1:27017','127.0.0.1:27018','127.0.0.1:27019'];" \
  $DMK_HOME/js/csrs_initiate.js

# Initiate shard 1 with an arbiter
mongosh 127.0.0.1:27020 \
  --eval "var rsName='rs1'; var rsHosts=['127.0.0.1:27020','127.0.0.1:27021']; var arbiterHost='127.0.0.1:27022';" \
  $DMK_HOME/js/rs_initiate.js

# Add all shards from a mongos router
mongosh 127.0.0.1:27030 \
  --eval "var shards=['rs1/127.0.0.1:27020,127.0.0.1:27021','rs2/127.0.0.1:27023,127.0.0.1:27024','rs3/127.0.0.1:27026,127.0.0.1:27027'];" \
  $DMK_HOME/js/sh_addshards.js

New --check-status option in dmk_db_ctl.py

The dmk_db_ctl.py script now accepts a --check-status flag. When provided, it polls the instance after the start, stop, or restart command to confirm the instance has reached the expected state. This was the default behaviour until now, which led to unwanted errors when setting up DMK.

dmk_db_ctl.py -i mdb01 -a start --check-status
dmk_db_ctl.py -i mdb01 -a stop --check-status
dmk_db_ctl.py -i mdb01 -a restart --check-status

Without this flag, the script returns immediately after issuing the command. It should be noted that when using the restart option, the status will be checked after stopping the instance. Otherwise, if the instance is still running when the start was issued, the restart could silently succeed without actually restarting anything.

Changes made to dmk_status.py

Colored status output. STARTED is now displayed in green, STOPPED in red — making it easier to spot unhealthy instances at a glance. To disable colors (for log files, CI pipelines, or terminals that don’t support ANSI), set COLORED_STATUS=no in your local configuration file (dmkl) by uncommenting the following line:

# var::COLORED_STATUS::=::nowarn::no::  # Disable colored status output in dmk_status.py (yes/no)

CSV output. The table view now supports CSV export via the --csv option, useful for piping output to other tools or exporting to a file:

# Export status as semicolon-delimited CSV
> dmk_status.py -t -a --csv ";"
Instance;Status;Version;Port;Bind IPs
mdb01;STARTED;8.0.26;27417;127.0.0.1
mdb02;STARTED;8.0.26;27418;127.0.0.1
mdbcfg01;STARTED;8.0.26;27017;127.0.0.1
mdbmgs01;STARTED;8.0.26;27030;127.0.0.1

The session view (sess) supports the same option.

Process-only check. The new -p / --process option performs a quick process presence check without connecting to MongoDB. This is faster and does not require credentials, making it useful for lightweight health checks:

dmk_status.py -t -a --process

Improved version display. The -mv option now shows all installed MongoDB versions side by side in a single table instead of separate blocks, making it much easier to compare binary versions across multiple installations.

Session display refactoring. Session data gathering was moved into the instance class, and the display logic (including filtering and formatting) was centralized in dmk_status.py. This also fixes the --include-inactive flag, which was not working correctly in the previous release.

New update_mongo_conf.py script

The new update_mongo_conf.py script modifies MongoDB configuration file parameters from the command line using dot-notation for nested keys, without manually editing YAML:

# Enable TLS on an existing instance
update_mongo_conf.py mdb01 net.tls.mode requireTLS
update_mongo_conf.py mdb01 net.tls.certificateKeyFile /u01/app/mongodb/admin/mdb01/secret/server.pem

New set_uri.py script

The new set_uri.py script generates and persists a MONGO_URI in the local DMK configuration for a given instance. It supports both standalone and replica set connection strings:

# Standalone
set_uri.py mdb01 sa 127.0.0.1:27017

# Replica set
set_uri.py mdbrs01 rs 127.0.0.1:27411,127.0.0.1:27412,127.0.0.1:27413 rs1

The resulting URI is stored in the local configuration file and available as $MONGO_URI in the instance context.

Improvements to dmk_dbcreate.py

dmk_dbcreate.py now reads the [instance_name] section from the local DMK configuration file when creating an instance, so instance-specific variables defined in your local config are picked up automatically. Empty values for MONGO_BACKUP, MONGO_JOURNAL, and MONGO_LOG are also now accepted — directories are only created for paths that are explicitly set.

The storageEngine key was removed from all YAML configuration templates. WiredTiger is the only storage engine available in MongoDB Community Edition, making the field redundant.

A new mongod.service.template_fork systemd unit (Type=simple, delegating start/stop to dmk_db_ctl.py) was also added, alongside the existing mongod.service.template that runs mongod in the foreground. dmk_dbcreate.py generates the mongod unit for mongod instances and the mongos unit for routers.

Installation improvements

Auto-creation of .bash_profile. If the .bash_profile of the mongodb user does not contain the DMK block, DMK now creates it automatically from the template on first sourcing. If the file already exists with a DMK block, only that block is updated in place (with a backup created first). This removes a manual step from first-time installation.

DMK temp files. The dmk_env.sh file used to source instance environments is now written to $DMK_HOME/tmp/ rather than /tmp, keeping temporary DMK files within the DMK directory tree.

Security improvements

The default bindIp in all DMK configuration templates was changed from 0.0.0.0 to 127.0.0.1. This is a safer default for freshly created instances that limits network exposure from the start.

A new MONGO_TLS_CERTIFICATE_KEY_FILE environment variable allows specifying a custom client certificate for DMK connections. This is useful for instances secured with serverAuth-only TLS, where the default certificate path may not be the right client certificate to use.

New TLS templates for replica sets (mongo_conf_rs_tls.yaml, mongo_conf_rs_tls_fork.yaml, mongo_conf_rs_keyfile.yaml) were also added.

Aliases and environment variables added in DMK 2.5.0

New aliases in DMK 2.5.0:

• mglogrotate: rotate the MongoDB log file (db.adminCommand({ logRotate: 1 })).
• certrotate: rotate the TLS certificate of an instance (db.rotateCertificates()).

New environment variables:

• COLORED_STATUS: set to no to disable colored status output in dmk_status.py. Enabled by default.
• MONGO_TLS_CERTIFICATE_KEY_FILE: custom client TLS certificate path for DMK connections.
• MONGO_URI: the MongoDB connection URI, now easily managed with set_uri.py.
• INSTANCE_NAMING_CONVENTION: the regular expression used to validate instance names. This variable is now optional — if not set, no naming convention is enforced.

Bug fixes

A few notable bugs were fixed in this release:

• Quoted values in the MongoDB configuration file (viewed with vic / dmkc) could cause the DMK environment to fail loading due to unescaped double quotes. This is now handled correctly.
• Restart could silently succeed without actually restarting the instance if it was still running. The stop phase now always waits for the instance to be fully stopped first.
• --include-inactive flag in the session view was not working correctly and has been fixed.

Upgrading from an earlier version of DMK

If you are upgrading from DMK 2.4.0, replace the DMK folder with the new one:

cd /u01/app/mongodb/local
mv dmk .dmk_old
unzip /path/to/dmk_mongodb-2.5.0.zip
dmk

If you are upgrading from DMK 2.3.x or earlier, read the release notes of DMK 2.4.0 first.

L’article MongoDB DMK 2.5.0: Sharding support and new features est apparu en premier sur dbi Blog.

The DMK is a set of standardized tools that aims at easing the work of DBAs by having dbi’s best practices embedded in common scripts across all the database servers of an organization.

dbi services provides the DMK (Database Management Kit) to its customers for multiple technologies: Oracle, Postgres, MongoDB, etc. This toolkit is provided free of charge to all clients who work with dbi services on a consulting project.

New features of the MongoDB DMK in 2.4.0

Changes made to dmk_status.py script

• The table view (accessible with dmk_status.py -t, or the tsta alias) now supports sorting by any column. Use the -o option followed by the column name to specify the sorting column. For instance, use -o='port' to sort by port number. Sorting is ascending by default, but you can add a negative sign - before the column name for descending order, for instance -o='-port'. In addition to this, you can specify multiple columns for sorting, like -o='-version,state' to sort first by version and then by state.

mongodb@dmkmongo:/home/mongodb/ [mdb01] sta -o="-version"

Instance             | State                | Version | Port  | Bind IPs
-------------------------------------------------------------------------------
mdb01                | STARTED              | 8.0.16  | 27017 | 0.0.0.0
mdb02                | STARTED              | 8.0.16  | 27016 | 0.0.0.0
mdbrs01              | STARTED              | 8.0.15  | 27101 | 0.0.0.0

• The session view (dmk_status.py -s, or sess alias) now supports filtering. You can filter with multiple criteria and combine them as needed. Here are the available filters:
  • Minimum runtime (--min-runtime <seconds>)
  • Client IP (--client <ip_address>)
  • User filtering (--user <username>)
  • Application name (--app <application_name>)
  • Database and collection (--ns <namespace>)

• You can also add the --show-query option to display the query for each session, which can be very useful in combination with the filters to quickly identify problematic queries. For instance, --min-runtime 10 --show-query will show all sessions that have been running for more than 10 seconds along with their queries.

Here is an example of session display filtering with the --app option:

mongodb@dmkmongo:/home/mongodb/ [mdb01] sess --app DataGrip
2026-02-13 13:18:30 | INFO | Total sessions: 6, Active sessions: 6

--------------------------------------------------  Active MongoDB Sessions  --------------------------------------------------

Filters applied: app: DataGrip, active only

ID              USER                 CLIENT                 OP         NS                   COMMAND        TIME       APP
conn7           -                    192.168.105.1:60423    command    admin.$cmd           hello          7.82s      DataGrip
conn8           -                    192.168.105.1:60424    command    admin.$cmd           hello          7.83s      DataGrip
conn9           -                    192.168.105.1:60425    command    admin.$cmd           hello          7.83s      DataGrip

-------------------------------------------------------------------------------------------------------------------------------

When viewing the status of a single instance with lsta (or dmk_status.py -l), the list of databases is now displayed.

mongodb@dmkmongo:/home/mongodb/ [mdb01] lsta
--------------------------   dbi services   ------------------------------

INSTANCE                 : mdb01
STATUS                   : STARTED
DATABASES                : dbapp01, dbapp02
VERSION                  : 8.0.16
BIND                     : 0.0.0.0
PORT                     : 27017
REPLICA SET STATUS       : No replication
CONFIGURATION FILE       : /u01/app/mongodb/admin/mdb01/etc/mdb01.conf
BIN PATH                 : /u01/app/mongodb/product/8.0.16/bin
DATA PATH                : /u02/mongodb/data/mdb01
LOG FILE                 : /u04/mongodb/log/mdb01/mdb01.log
BACKUP PATH              : /u90/mongodb/backup/mdb01

--------------------------------------------------------------------------

New replicat set features

• The rsta alias (equivalent of dmk_status.py -r) was improved to display more information.

mongodb@dmkmongo:/home/mongodb/ [mdbrs02] rsta

--------------------------------------------------  Replica Set Status Summary  --------------------------------------------------

INSTANCE                 : mdbrs02
REPLICA SET NAME         : rs01
DATE                     : 2026-02-13T12:21:43.755Z
MEMBER STATE             : SECONDARY
MEMBERS                  :
    NAME                  | STATE     | UPTIME | LAG  | OPTIME              | HEALTH | PRIORITY | SYNC FROM
    192.168.105.200:27017 | SECONDARY | 1m5s   | 0.0s | 2026-02-13T12:21:38 | UP     | 1        | 192.168.105.201:27017
    192.168.105.201:27017 | SECONDARY | 1m5s   | 0.0s | 2026-02-13T12:21:38 | UP     | 1        | 192.168.105.202:27017
    192.168.105.202:27017 | PRIMARY   | 48s    | 0.0s | 2026-02-13T12:21:38 | UP     | 1        |

----------------------------------------------------------------------------------------------------------------------------------

• A stepdown alias was added to allow stepping down the primary directly from the DMK command line. Its definition is rather simple: alias stepdown='msp --eval "rs.stepDown()"'.

Improvements to the set_local_dmk_config.py script

The set_local_dmk_config.py script now supports setting the local DMK configuration for custom instances by adding categories. Its usage doesn’t change, and you can find more about it in the Environment variables page of the DMK documentation.

Better support for instances managed with systemd

DMK better supports MongoDB instances that are managed with systemd. The dmk_db_ctl.py script will automatically detect if an instance is managed with systemd and use systemctl commands to start, stop, or restart the instance. This allows for better integration with the operating system and provides a more reliable management of MongoDB instances.

• Two new aliases were added to access systemd unit files of instances: cdsys to access the folder containing the unit file and visys to view the content of the unit file.
• A MONGO_SERVICE_NAME variable was added to the configuration, which allows specifying the name of the systemd service for each instance. This is useful when the service name doesn’t follow a standard pattern based on the instance name. By default, MONGO_SERVICE_NAME is set to mongod_${MONGO_INSTANCE}.service, but you can overwrite it in your local configuration file if needed.

New aliases and environment variables

Aliases added in DMK 2.4.0:

• cdp: alias to access the $MONGO_BASE/product folder.
• cdbin: alias to access the MongoDB binaries folder of an instance.
• dmkjs: alias to access the DMK JavaScript folder. It is also available as a variable (dmkjs=$DMK_HOME/js).
• cdsys: alias to access the systemd unit files folder.
• visys: alias to view the systemd unit file of an instance.
• stepdown: alias to step down a primary instance with the rs.stepDown() command.
• mgstatus: alias to show the status of the current instance. It has the same output as the lsta alias, but it was added to work with the other mgstop, mgstart and mgrestart aliases.

Environment variables added in DMK 2.4.0:

• MONGO_SERVICE_NAME environment variable to specify the name of the systemd service for each instance. By default, it is set to mongod_${MONGO_INSTANCE}.service, but you can overwrite it in your local configuration file if needed.
• MONGO_CRED variable to specify the path to the credential file of an instance. It is set by default to ${MONGO_BASE}/admin/${MONGO_INSTANCE}/secret/cred.yaml, but you can overwrite it in your local configuration file if needed. This variable is used in the msp alias to automatically provide credentials when connecting to an instance with mongosh. More specifically, you can now set MONGO_CRED in the [INSTANCE] group in your local configuration file to have the msp alias automatically use the same credentials for all instances, which is very useful when managing multiple instances with the same credentials.

Upgrading from an earlier version of DMK

• If you are upgrading from 2.2.0 or earlier, please follow the instructions detailed in the release notes of DMK 2.3.
• If you are upgrading from DMK 2.3, you just have to replace the current dmk folder with the new one.

# Go to the DMK folder
cd /u01/app/mongodb/local

# Rename the current dmk folder with a hidden folder
mv dmk .dmk_old

# Unzip the new version of the DMK
unzip /path/to/dmk_mongodb_2.4.0.zip

# Reload DMK
dmk

L’article MongoDB DMK 2.4.0 : new features est apparu en premier sur dbi Blog.

dbi services provides the DMK (Database Management Kit) to its customers for multiple technologies: Oracle, Postgres, MongoDB, etc. This toolkit is provided free of charge to all clients who work with dbi services on a consulting project.

The DMK is a set of standardized tools aiming at easing the work of DBAs, by having dbi’s best practices embedded in common scripts across all the database servers of an organization.

New features of the MongoDB DMK

Rewriting of the project

The most significant changes in the MongoDB DMK is the rewriting of all old Perl scripts into new Python scripts. On top of being more adapted to the MongoDB ecosystem, these will improve modularity for customers wanting to write their own packages.

It means that all utility scripts are now named .py instead of .sh, and apart from new features that have been added, the basic behavior stays the same for all of them.

DMK configuration file

Before release 2.3.0, only one configuration file existed in $DMK_HOME/etc. There is now a second configuration file in ~/.DMK/dmk.conf.local, which will overwrite default configuration options. See the GitBook section on Using DMK for more information.

New default directories and more versatility

The Optimal Flexible Architecture (OFA) has new recommendations. Specifically, the new default architecture is the following:

• /u01 for binaries and admin folders
• /u02 for database files
• /u03 for journal files
• /u04 for log files
• /u90 for backup files

Even though dbi suggests OFA as a good standard for MongoDB installations, we know that a lot of legacy installations will not use this kind of architecture. This is why the DMK is now more versatile, and with the use of the local configuration file described above, it has never been easier to adapt the DMK to your needs.

New aliases and environment variables

Some aliases were changed in this release, others were added. See Environment Variables and Aliases in the documentation for more information.

• mgstart, mgstop, mgrestart are new aliases to manage a MongoDB instance.
• vic now opens the MongoDB instance configuration file.
• vilst now opens the $DMK_HOME/etc/mongodb.lst file.
• sta, lsta, tsta, rsta are new aliases for instance status display.
• vil, cdl, tal are new aliases to view, access and tail log files of MongoDB instances.
• dmkc opens DMK default configuration file.
• dmkl opens DMK local configuration file, which overrides the default configuration file.

Other changes

• A script named set_local_dmk_config.py was created to automate local configuration file changes. See Environment Variables for more details.
• Backups are no longer compressed by default, and the option to compress them has been added to the dmk_dbbackup.py script.
• And of course, corrections of bugs.

Installing the DMK for the first time

Installing the DMK is always fairly easy. If you follow the OFA, just unzip the package and source dmk.sh.

[root@vm00 ~]$ su - mongodb
[mongodb@vm00 ~]$ unzip -oq dmk_mongodb-2.3.1.zip -d /u01/app/mongodb/local
[mongodb@vm00 ~]$ . /u01/app/mongodb/local/dmk/bin/dmk.sh
2025-12-04 10:03:48 | INFO | DMK_HOME environment variable is not defined.
2025-12-04 10:03:48 | INFO | First time installation of DMK.
2025-12-04 10:03:48 | INFO | DMK has been extracted to /u01/app/mongodb/local/dmk
2025-12-04 10:03:48 | INFO | Using DMK_HOME=/u01/app/mongodb/local/dmk
2025-12-04 10:03:48 | INFO | Default configuration file '/u01/app/mongodb/local/dmk/etc/dmk.conf.default' does not exist. Creating it.
2025-12-04 10:03:48 | INFO | Copying template file '/u01/app/mongodb/local/dmk/templates/etc/dmk.conf.unix' to '/u01/app/mongodb/local/dmk/etc/dmk.conf.default'
2025-12-04 10:03:48 | INFO | Local configuration file does not exist. Creating it.
2025-12-04 10:03:48 | INFO | Copying template file '/u01/app/mongodb/local/dmk/templates/etc/dmk.conf.local.template' to '/home/mongodb/.dmk/dmk.conf.local'
2025-12-04 10:03:48 | INFO | Creating symlink '/u01/app/mongodb/local/dmk/etc/dmk.conf.local' to '/home/mongodb/.dmk/dmk.conf.local'
2025-12-04 10:03:48 | WARNING | MONGO_BASE environment variable is not set. Trying to retrieve it from DMK_HOME (/u01/app/mongodb/local/dmk).
2025-12-04 10:03:48 | WARNING | MONGO_BASE set to '/u01/app/mongodb' based on DMK_HOME location.
2025-12-04 10:03:48 | WARNING | If you're running DMK for the first time, you can ignore these warnings.
2025-12-04 10:03:48 | WARNING | Otherwise, please set MONGO_BASE in /home/mongodb/.DMK before sourcing DMK.
2025-12-04 10:03:48 | WARNING | File '/u01/app/mongodb/etc/mongodb.lst' does not exist. Creating an empty file.
2025-12-04 10:03:48 | INFO | Creating DMK source file at '/home/mongodb/.DMK' with the following content:
2025-12-04 10:03:48 | INFO | DMK_HOME=/u01/app/mongodb/local/dmk
2025-12-04 10:03:48 | INFO | PYTHON_BIN=/usr/bin/python3
2025-12-04 10:03:48 | INFO | MONGO_BASE=/u01/app/mongodb
2025-12-04 10:03:48 | WARNING | Please make sure to source the .DMK file in your shell profile (e.g., .bash_profile).
2025-12-04 10:03:48 | WARNING | An example is provided at /u01/app/mongodb/local/dmk/templates/profile/dmk.mongodb.profile

If you don’t follow the OFA, you should define the following mandatory variables before running the DMK, inside the /home/mongodb/.DMK file:

• DMK_HOME: path to the DMK main folder
• PYTHON_BIN: path to the Python binaries (3.6+ necessary, which is the default for Linux 8-like platforms)
• MONGO_BASE

[root@vm00 ~]$ su - mongodb
[mongodb@vm00 ~]$ echo "DMK_HOME=/u01/app/mongodb/local/dmk" > ~/.DMK
[mongodb@vm00 ~]$ echo "PYTHON_BIN=/usr/bin/python3" >> ~/.DMK
[mongodb@vm00 ~]$ echo "MONGO_BASE=/u01/app/mongodb" >> ~/.DMK

[mongodb@vm00 ~]$ cat ~/.DMK
export DMK_HOME=/u01/app/mongodb/local/dmk
export PYTHON_BIN=/usr/bin/python3
export MONGO_BASE=/u01/app/mongodb

Loading DMK at login

If you want the DMK to be loaded when logging in, you should add the following code block to the .bash_profile of the mongodb user:

# BEGIN DMK BLOCK
if [ -z "$DMK_HOME" ]; then
  if [ -f "$HOME/.DMK" ]; then
    . "$HOME/.DMK"
  else
    echo "$HOME/.DMK file does not exist"
    return 1
  fi
fi

# Launched at login
. ${DMK_HOME}/bin/dmk.sh && ${PYTHON_BIN} ${DMK_HOME}/bin/dmk_status.py --table --all
# END DMK BLOCK

After this, you can just log in again. The installation is complete !

Migrating from a former version of the DMK

If you already have the MongoDB DMK installed on your systems, there are a few more steps to take for this specific upgrade, because we switched from old Perl libraries to Python.

You first need to adapt the .DMK file, as described in the installation steps.

[mongodb@vm00 ~]$ cat ~/.DMK
export DMK_HOME=/u01/app/mongodb/local/dmk
export PYTHON_BIN=/usr/bin/python3
export MONGO_BASE=/u01/app/mongodb

Then, move the former DMK folder and unzip the new version of the DMK. The old DMK should be a hidden directory, otherwise DMK will consider it as a custom package !

mongodb@vm00:/home/mongodb/ [DUMMY] cd /u01/app/mongodb/local/
mongodb@vm00:/u01/app/mongodb/local/ [DUMMY] ls -l
drwxrwx---. 10 mongodb mongodb 118 Jul  1 04:34 dmk
mongodb@vm00:/u01/app/mongodb/local/ [DUMMY] mv dmk .dmk_old
mongodb@vm00:/u01/app/mongodb/local/ [DUMMY] unzip /u01/app/mongodb/artifacts/dmk_mongodb-2.3.1.zip
mongodb@vm00:/u01/app/mongodb/local/ [DUMMY] ls -ail
100690250 drwxrwx---.  8 mongodb mongodb  96 Jul  1 04:24 dmk
 33554663 drwxrwx---. 10 mongodb mongodb 118 Jul  1 04:34 .dmk_old

Update your .bash_profile to remove all traces of the former DMK loading mechanism. Here is an example of the minimal DMK block in the template file:

# BEGIN DMK BLOCK
if [ -z "$DMK_HOME" ]; then
    if [ -f "$HOME/.DMK" ]; then
        . "$HOME/.DMK"
    else
        echo "$HOME/.DMK file does not exist. It is needed to source DMK at login. Run '. <DMK_HOME>/bin/dmk.sh' or 'source <DMK_HOME>/bin/dmk.sh' to source DMK manually this time."
        return 1
    fi
fi

# Launched at login
. ${DMK_HOME}/bin/dmk.sh && ${PYTHON_BIN} ${DMK_HOME}/bin/dmk_status.py --table --all
# END DMK BLOCK

Last but not least, you will have to customize your local DMK configuration file ~/.dmk/dmk.conf.local. You can use the set_local_dmk_config.py script to help yourself with the modifications.

mongodb@vm00:/u01/app/mongodb/admin/ [mdb01] set_local_dmk_config.py INSTANCE MONGO_JOURNAL "\${MONGO_DATA_ROOT}/\${MONGO_INSTANCE}/journal"
Backup created: /home/mongodb/.dmk/dmk.conf.bak_20251024_084959
Updated MONGO_JOURNAL in [INSTANCE]
Old value: var::MONGO_JOURNAL::=::nowarn::"${MONGO_JOURNAL_ROOT}/${MONGO_INSTANCE}"::
New value: var::MONGO_JOURNAL::=::nowarn::"${MONGO_DATA_ROOT}/${MONGO_INSTANCE}/journal"::
Use 'dmkc' and 'dmkl' aliases to quickly view default and local configuration files.

For any questions regarding the MongoDB DMK, take a look at the documentation or feel free to contact me.

L’article MongoDB DMK 2.3: new features est apparu en premier sur dbi Blog.

• Encryption in transit
• Encryption at rest
• Encryption in use

Among these, encryption in transit is fundamental : it protects data as it moves between your application and the database. In MongoDB, this is achieved through TLS (Transport Layer Security), which ensures that communication remains private and secure. You have two options when it comes to using TLS for your database :

• Using TLS for encryption only.
• Using TLS both for encryption and authentication to the database.

Setting up TLS for encryption

Create a Certificate Authority

We’ll first create a Certificate Authority. These certificates will be self-signed, which is fine for testing, but you shouldn’t use self-signed certificates in a production environment ! On Linux, use the openssl library to generate the certificates.

openssl req -newkey rsa:4096 -nodes -x509 -days 365 -keyout ca.key -out ca.pem -subj "/C=CH/ST=ZH/L=Zurich/O=dbi/OU=MongoDBA/CN=vm.domain.com"

Here is a description of some important parameters of the commands :

• -newkey rsa:4096 : Generates a new private key and a certificate request using RSA with a 4096-bit key size.
• -nodes : Skips password encryption of the private key. Without it, OpenSSL would prompt you to set a passphrase.
• -x509 : Generates a self-signed certificate. x509 is supported by MongoDB.
• -days 365 : Validity of the certificate in days.
• -keyout ca.key : Filename for the private key.
• -out ca.pem : Filename for the certificate.
• -subj "..." : Provides the subject’s Distinguished Name (DN). If you don’t specify it, OpenSSL will prompt for each field.

Create a Server Certificate

Then, we’ll create the server certificate for the MongoDB instance. In the openssl-server.cnf file below, you should change the req_distinguished_name fields with what you used while creating the Certificate Authority, and replace vm.domain.com by the name of your machine.

If you only have an IP and no DNS entry for your VM, use IP.1 instead of DNS.1 in the alt_names section.

cat > openssl-server.cnf <<EOF
[ req ]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no

[ req_distinguished_name ]
C = CH
ST = ZH
L = Zurich
O = dbi
OU = MongoDBA
CN = myVM

[ v3_req ]
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = myVM
EOF

Then, generates the certificate with these commands :

openssl req -newkey rsa:4096 -nodes -keyout mongodb-server.key -out mongodb-server.csr -config openssl-server.cnf

openssl x509 -req -in mongodb-server.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out mongodb-server.crt -days 365 -extensions v3_req -extfile openssl-server.cnf

cat mongodb-server.key mongodb-server.crt > mongodb-server.pem

Create a Client Certificate

Finally, we’ll create a client certificate. The process is the same, with a few tweaks :

• OU should be different from the one from the server certificate. It is not mandatory for the communication, but it will be for the authentication if you decide to enable it.
• CN should also be different.
• extendedKeyUsage should be set with clientAuth instead of serverAuth.

cat > openssl-client.cnf <<EOF
[ req ]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no

[ req_distinguished_name ]
C = CH
ST = ZH
L = Zurich
O = dbi
OU = MongoDBAClient
CN = userApp

[ v3_req ]
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth
EOF

The creation of the certificate is the same.

openssl req -newkey rsa:4096 -nodes -keyout mongodb-client.key -out mongodb-client.csr -config openssl-client.cnf

openssl x509 -req -in mongodb-client.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out mongodb-client.crt -days 365 -extensions v3_req -extfile openssl-client.cnf

cat mongodb-client.key mongodb-client.crt > mongodb-client.pem

MongoDB Configuration Change

Make sure to set permissions correctly for your certificates.

chmod 600 ca.pem mongodb-server.pem mongodb-client.pem
chown mongod: ca.pem mongodb-server.pem mongodb-client.pem

Now, you can change your MongoDB configuration file to include the certificates. Simply add the net.tls part to your mongod.conf file.

net:
  bindIp: yourIP
  port: 27017
  tls:
    mode: requireTLS
    certificateKeyFile: /path/to/mongodb-server.pem
    CAFile: /path/to/ca.pem

You can now restart your MongoDB instance with systemctl restart mongod (or whatever you’re using), and then try the connection to your instance for your client. Of course, the port mentioned in the net.port field of your configuration file shouldn’t be blocked by your firewall.

> mongosh --host myVM --port 27017 --tls --tlsCertificateKeyFile mongodb-client.pem --tlsCAFile ca.pem
Current Mongosh Log ID:	682c9641bbe4593252ee7c8c
Connecting to:		mongodb://vmIP:27017/?directConnection=true&tls=true&tlsCertificateKeyFile=Fclient.pem&tlsCAFile=ca.pem&appName=mongosh+2.5.1
Using MongoDB:		8.0.9
Using Mongosh:		2.5.1

For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/

test>

You’re now connected to your MongoDB instance through TLS ! And if you’re not using the certificate, the requireTLS mode prevents the connection from being established, and generated these error messages in your logs :

{"t":{"$date":"2025-05-21T04:43:55.277+00:00"},"s":"I",  "c":"EXECUTOR", "id":22988,   "ctx":"conn52","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"The server is configured to only allow SSL connections"},"remote":"IP:50100","connectionId":52}}

If you want to learn more about MongoDB logs, I wrote a blog on this topic: MongoDB Log Analysis : A Comprehensive Guide.

Setting up TLS authentication

Now that you’re connected, we will set up authentication so that you can be connected as a specific user to MongoDB. Using the already established connection, create a user in the $external database. Each client certificate that you create can be mapped to one MongoDB user. Retrieve the username that you will use :

> openssl x509 -in mongodb-client.pem -inform PEM -subject -nameopt RFC2253 | grep subject
subject=CN=userApp,OU=MongoDBA,O=dbi,L=Zurich,ST=ZH,C=CH

And then create the user in the $external database, using the existing MongoDB connection :

test> db.getSiblingDB("$external").runCommand({
  createUser: "CN=userApp,OU=MongoDBA,O=dbi,L=Zurich,ST=ZH,C=CH",
  roles: [
    { role: "userAdminAnyDatabase", db: "admin" }
  ]
});

To check that everything works as intended, you can try to display collections in the admin database. For the moment, there is no error because you have all the rights to do it (no authorization is enforced).

test> use admin
switched to db admin
admin> show collections;
system.users
system.version

You can now edit the MongoDB configuration file by adding the net.tls.allowConnectionsWithoutCertificates set to true, and the security.authorization flag set to enabled. The mongod.conf file should look like this :

net:
  bindIp: X.X.X.X
  port: XXXXX
  tls:
    mode: requireTLS
    certificateKeyFile: /path/to/mongodb-server.pem
    CAFile: /path/to/ca.pem
    allowConnectionsWithoutCertificates: false

security:
  authorization: enabled

After restarting with systemctl restart mongod, you can now connect again. If you use the same command as before, you will log in without authentication, and get the error below whenever you try to do anything :

MongoServerError[Unauthorized]: Command listCollections requires authentication

So you should now connect via this command :

mongosh --host vmIP --port 27017 --tls --tlsCertificateKeyFile mongodb-client.pem --tlsCAFile ca.pem --authenticationDatabase '$external' --authenticationMechanism MONGODB-X509

If you want to show the admin collections, you will now get an error, because your user only has the userAdminAnyDatabase role granted (this role was chosen during the user creation, see above).

admin> show collections
MongoServerError[Unauthorized]: not authorized on admin to execute command { listCollections: 1, filter: {}, cursor: {}, nameOnly: true, authorizedCollections: false, lsid: { id: UUID("9c48b4c4-7702-49ce-a97c-52763b2ad6b3") }, $db: "admin" }

But it’s fine, you can grant yourself more roles (readWriteAnyDatabase, for instance) and create new users if you want.

The communication between the client and the server is now fully secured. Congratulations !

MongoServerError[BadValue]

Side note : if you ever encounter this error:

MongoServerError[BadValue]: Cannot create an x.509 user with a subjectname that would be recognized as an internal cluster member

… make sure to follow the RFC-2253 standards. For instance, you could have this error if one of the field is too long. Also, as a reminder, the client certificate should have a different Distinguished Name (DN) than the server certificate (see documentation for more information).

L’article Setting up TLS encryption and authentication in MongoDB est apparu en premier sur dbi Blog.

In response to this, Oracle (and others) use a not-so-native adaptation of SQL to query JSON documents. And if you’ve ever had to deal with these SQL-based JSON queries, you may have found yourself thinking (in hindsight) that querying JSON documents with MQL (MongoDB Query Language) wasn’t that painful after all. And rightly so : JSON documents are very different from relational tables, and there is no easy way to continue using SQL without learning some new syntax.

Oracle likely came to the same conclusion, and offers an API dedicated to MongoDB natives. The idea is simple : to ease the work of developers and database administrators when deciding to migrate from MongoDB to Oracle (and the other way around ?…)

On Oracle Cloud Infrastructure (OCI), this means being able to connect to Autonomous Databases (ADB). As a reminder, you have two of them included in the Always Free Tier of OCI, so you can play around with this feature for free before deciding to migrate to the cloud.

Setting up the Oracle MongoDB API on OCI

When creating an Autonomous Database, you can decide on multiple workloads types, including a JSON workload. However, the MongoDB API does not strictly require the JSON workload type to work.

However, the network access setting of your Autonomous Database must be non-default with one of the following options enabled :

• Secure access from allowed IPs and VCNs only
• Private endpoint access only

For instance, when using the Secure access from allowed IPs and VCNs only option, you can add any IP address to the Access Control List (ACL), including your own.

After creating the ADB (Autonomous Database), you can check in the Tool configuration tab the MongoDB API is enabled (it should be, by default).

Then you can go in Database actions > Database Users to either create a user or modify an existing one, with the following privileges : CREATE SESSION, CREATE RESOURCE (default), and the SODA_APP role. After creation, you should enable REST as shown below. This will allow the API to work on that particular schema.

Back on the Tool configuration tab of your ADB instance, you’ll find the Public access URL. Copy it, this will be your connection string when using the API.

Connection String Example

For this to work, you have to replace the user and password in the connection string. For instance, if your user is called USER_DB, and your password is userpwd, then you would use this connection string (without the brackets). Make sure to escape any special character in the password when doing so (see documentation).

mongodb://user_db:userpwd@A49E7CHE9B320E6-PCCVX8PAE2NZPVBQ.adb.eu-zurich-1.oraclecloudapps.com:27017/user_db?authMechanism=PLAIN&authSource=$external&ssl=true&retryWrites=false&loadBalanced=true'

Use this connection string with mongosh or any tool provided by MongoDB. With mongosh, you would connect with this command :

mongosh 'mongodb://user_db:userpwd@A49E7CHE9B320E6-PCCVX8PAE2NZPVBQ.adb.eu-zurich-1.oraclecloudapps.com:27017/adb_admin?authMechanism=PLAIN&authSource=$external&ssl=true&retryWrites=false&loadBalanced=true'

You are now connected to the Oracle ADB instance through the mongosh utility.

Viewing Oracle tables with the API

If you connect to an existing schema with tables, you will notice that running show collections (or show tables) doesn’t return anything. In Oracle 19c, only collections/tables created through the API appear in this list. Standard tables created with a CREATE TABLE SQL statement won’t be displayed.

Included and excluded features in the API

Once connected, you can query the different databases (schemas, in Oracle terms) of your instance. However, unlike in MongoDB, you can’t administer the instance through mongosh.

You can look at the documentation to check the features included in the API, but here is an extract of the most notable changes :

• All the most basic aggregated pipelines are available in the API, but more complex ones like $currentOp or $merge are not yet supported.
• Other aggregation operators, like $bucket, $documents or $sample are only available starting with Oracle 23ai.
• db.dropDatabase() doesn’t drop the Oracle schema. In fact, it doesn’t even delete everything in the schema. It only deletes all real collections inside of it. So if you have a traditional table that is not considered as a collection (not shown in show collections), it will not be deleted by this command.

Troubleshooting MongoServerError: Database connection unavailable

You might encounter the following error :

MongoServerError: Database connection unavailable. Ensure that the user exists and the schema is enabled for use with Oracle REST Data Services. A schema can be enabled by calling the PL/SQL procedure ORDS.ENABLE_SCHEMA

If you see this message, you can either follow the procedure or remember to enable REST directly on the OCI Database Actions panel, as shown above.

In short, Oracle’s MongoDB API provides an interesting bridge between the relational and document worlds. Even though some MongoDB features are not supported yet, the API offers a straightforward way to connect MongoDB and Oracle Autonomous Database. It makes it ideal for testing, for migration or even for hybrid setups. For developers used to MQL, it can significantly ease the transition to Oracle’s ecosystem.

L’article Using Oracle API for MongoDB on OCI est apparu en premier sur dbi Blog.

Log format overview

Inside /var/log/mongodb/mongod.log (or a custom path if you set one), a typical log entry looks like this (shortened, for readability) :

{"t":{"$date":"2025-03-06T14:54:28.298+01:00"},"s":"I",  "c":"CONTROL",  "id":8423403, "ctx":"initandlisten","msg":"mongod startup complete","attr":{"Summary of time elapsed":{"Startup from clean shutdown?":false,"Statistics":{"Set up periodic runner":"0 ms","Set up online certificate status protocol manager":"0 ms",[...],"Start transport layer":"0 ms","_initAndListen total elapsed time":"626 ms"}}}}

At first glance, it’s pretty difficult to extract the essential information, but let’s see how the document looks like when it’s formatted (we’ll see how to do that later).

{
  "t": {
    "$date": "2025-03-06T14:54:28.298+01:00"
  },
  "s": "I",
  "c": "CONTROL",
  "id": 8423403,
  "ctx": "initandlisten",
  "msg": "mongod startup complete",
  "attr": {
    "Summary of time elapsed": {
      "Startup from clean shutdown?": false,
      "Statistics": {
        "Set up periodic runner": "0 ms",
        "Set up online certificate status protocol manager": "0 ms",
        [...] # lines hidden
        "Start transport layer": "0 ms",
        "_initAndListen total elapsed time": "626 ms"
      }
    }
  }
}

Here is a description of the main fields of the log document :

• t : Timestamp of the log entry.
• s : Severity code associated with the log entry (E for error, W for warning, I for information and D1 to D5 for debug).
• c : Category of the log entry. Most common categories are CONTROL, COMMAND, ELECTION, REPL (for replication) or NETWORK. An extensive list is available in the official MongoDB documentation.
• id : Unique log entry ID.
• ctx : Thread that generated the log.
• msg : Usually a short message describing the log.
• attr : Optional additional attributes.

This will help us when looking at the logs, first with mongosh.

Querying logs through mongosh

You can query logs inside the MongoDB shell called mongosh. To do so, use the getLog admin command :

db.adminCommand({ getLog: "global"}); // display all log entries

Another useful option is to view startup warnings, which will only display warning logs since last startup.

db.adminCommand({ getLog: "startupWarnings" }) // display startup warnings
{
  totalLinesWritten: 2,
  log: [
    '{"t":{"$date":"2025-03-07T08:32:41.005+01:00"},"s":"W",  "c":"NETWORK",  "id":5123300, "ctx":"initandlisten","msg":"vm.max_map_count is too low","attr":{"currentValue":65530,"recommendedMinimum":102400,"maxConns":51200},"tags":["startupWarnings"]}\n',
    '{"t":{"$date":"2025-03-07T08:32:41.005+01:00"},"s":"W",  "c":"CONTROL",  "id":8386700, "ctx":"initandlisten","msg":"We suggest setting swappiness to 0 or 1, as swapping can cause performance problems.","attr":{"sysfsFile":"/proc/sys/vm/swappiness","currentValue":60},"tags":["startupWarnings"]}\n'
  ],
  ok: 1
}

Even though this can sometimes be useful, it requires authenticated access to the database, and it only works when the mongod process is running. You won’t be able to use this method when the database crashes, for instance. Moreover, the logs are difficult to read.

Most of the time, you will be better served by the jq utility.

Navigating through the logs with jq

jq is a powerful utility that helps you navigate JSON documents, and even though it is not an official MongoDB product, you should always integrate it in your MongoDB deployments.

Prettify MongoDB logs

The first benefit of the jq command is to display MongoDB logs in a readable format :

> head -1 mongod.log | jq
{
  "t": {
    "$date": "2025-03-05T14:44:28.531+01:00"
  },
  "s": "I",
  "c": "CONTROL",
  "id": 23285,
  "ctx": "main",
  "msg": "Automatically disabling TLS 1.0"
}

Of course, a single line of log will now span multiple lines in the output. But thanks to the log structure explained above, we can write our first queries with jq to filter the results and only display what’s important.

I would definitely recommend to build aliases with the following commands in order to quickly access the information that you find more valuable in the logs.

Display error messages

By using the s field (severity), we can filter the logs to only display error messages. This is especially useful when failing to start a MongoDB instance.

jq 'select(.s == "E")' mongod.log

You can also include warnings by slightly modifying the command.

jq 'select(.s == "E" or .s == "W")' mongod.log

Filtering displayed fields

When selecting fields, you should pipe the jq filtering to a JSON object like this one :

{time: .t["$date"], message: .msg, error: .attr.error}

From now on, I will use the -c option to display the JSON in the compact format. Even in this format, logs can be readable when you select or exclude specific fields. Here, I want to select the .t["$date"], .msg and .attr.error fields. To improve the display, I will rename them :

> jq -c 'select(.s == "E") | {time: .t["$date"], message: .msg, error: .attr.error}' mongod.log
{"time":"2025-03-05T14:44:28.665+01:00","message":"WiredTiger error message","error":13}
{"time":"2025-03-05T14:44:28.665+01:00","message":"WiredTiger error message","error":13}
{"time":"2025-03-05T14:44:28.665+01:00","message":"WiredTiger error message","error":13}
{"time":"2025-03-06T10:17:07.383+01:00","message":"DBException in initAndListen, terminating","error":"Location28596: Unable to determine status of lock file in the data directory /var/lib/mongodb: boost::filesystem::status: Permission denied [system:13]: \"/var/lib/mongodb/mongod.lock\""}

Similarly, you can exclude a field with the del function. For instance, this will remove the message sub-field located inside the attr field.

jq 'del(.attr.message)' mongod.log

Errors and warnings grouped by message

To check for recurrent warnings or errors, you can pipe the jq output to group the results by msg content.

jq 'select(.s == "E" or .s=="W") | .msg' mongod.log | sort | uniq -c | sort -nr | head

Occurrences of each log severity

If you want to quickly look for the number of every severity, you can do so with the s field.

> jq '.s' mongod.log | sort | uniq -c
     10 "E"
      3 "F"
   1727 "I"
     88 "W"

View logs for specific log categories

As mentioned before, the category of the logs might be interesting to filter (only the replication logs, for instance).

jq -c 'select(.c == "REPL")' mongod.log

Filter logs by date

Whether you use log rotation or not, you might want to quickly access the last minutes/hours/days of logs. With the date utility, you can retrieve the most recent logs :

jq -c --arg since "$(date -d '10 minutes ago' +%Y-%m-%dT%H:%M:%S)" 'select(.t["$date"] >= $since)' mongod.log

Still filtering on the .t["$date"] field, you can also extract a portion of the logs :

jq -c 'select(.t["$date"] >= "2025-03-06T14:30:00" and .t["$date"] <= "2025-03-06T14:40:00")' mongod.log

Look for a specific pattern in the log

Of course, you can always use grep followed by jq to find a pattern in the logs : grep -i "pattern" mongod.log | jq

But if you want to look for a specific pattern inside a specific field, you can do so with the test function :

> jq -c 'select(.msg | test("failed to authenticate"; "i"))' mongod.log // (i option for case insensitivity)
{"t":{"$date":"2025-03-07T08:37:52.950+01:00"},"s":"I","c":"ACCESS","id":5286307,"ctx":"conn18","msg":"Failed to authenticate","attr":{"client":"xxx.xxx.xxx.xxx(ip):xxxxx(port)","isSpeculative":true,"isClusterMember":false,"mechanism":"SCRAM-SHA-256","user":"root","db":"admin","error":"AuthenticationFailed: SCRAM authentication failed, storedKey mismatch","result":18,"metrics":{"conversation_duration":{"micros":5091,"summary":{"0":{"step":1,"step_total":2,"duration_micros":62},"1":{"step":2,"step_total":2,"duration_micros":48}}}},"extraInfo":{}}}

Check for logs regarding connections to the MongoDB database

For filtering connections logs, search for the attr.remote field :

jq -c 'select(.attr.remote)' mongod.log

Analyzing slow queries with jq

Inside the Mongo shell, you can activate logging for slow queries with db.setProfilingLevel(1, <slowms>), with <slowms> being the threshold (in milliseconds) to log such queries.

Warnings related to slow queries in MongoDB :

• Once activated, the slow queries logging could slow down the database, so be very careful when activating it.
• There is a security threat when combining slow query logging and queryable encryption, since queries will not be encrypted in the mongod.log file.

Slow query logs look like this :

{
  "t": { "$date": "2024-03-06T12:34:56.789Z" },
  "s": "I",
  "c": "COMMAND",
  "id": 123,
  "ctx": "conn20",
  "msg": "Slow query",
  "attr": {
    "ns": "mydb.coll",
    "command": { "find": "coll", "filter": { "status": "active" } },
    "planSummary": "COLLSCAN",
    "keysExamined": 0,
    "docsExamined": 5000,
    "numYields": 0,
    "reslen": 2000,
    "locks": { "Global": { "acquireCount": { "r": 1 } } },
    "durationMillis": 150
  }
}

With this in mind, and with what we have already seen, you can filter the logs with the fields you want, like attr.durationMillis (duration of the query, in milliseconds), or attr.ns, which is the object on which the query is made.

For instance, if you want to retrieve slow queries above a given threshold (one second, in the example below) :

jq 'select(.attr.durationMillis >= 1000)' mongod.log

Or if you want to filter slow queries on a specific database mydb and collection coll :

jq 'select(.msg == "Slow query" and .attr.ns == "mydb.coll")' mongod.log

You can also select only queries that are run on a given database mydb :

jq 'select(.msg == "Slow query" and .attr.command["$db"] == "mydb")' mongod.log

MongoDB log jq query builder

On my personal blog, I designed a tool for MongoDB log analysis with jq. Feel free to check it out !

Conclusion

While being a bit complex at first sight, MongoDB logs are very useful if you know how to apprehend them. By leveraging the jq utility for advanced filtering, and combining it with monitoring tools, you can efficiently analyze logs and improve your efficiency as a DBA.

L’article MongoDB Log Analysis : A Comprehensive Guide est apparu en premier sur dbi Blog.

Last time, we have seen how to add a Redis plugin to Grafana in order to monitor and generate graphs and charts for your reports or analysis.

During this configuration you may encounter some issues or miss some steps that will end to an error message.

Today we will see some typical error message and who to fix it quickly.

Check if your plugin’s version installed on Grafana

First step is to have confirmation that your Redis plugin is installed and up to date.

• Go to Administration then Plugins

• Check if plugin is here

• As you see, plugin is here and installed

• Check the plugin version ( in case you are not able to debug it it may be useful for Redis support team )

Now you have the plugin version and also its compatibility with the Grafana version

( in our example it is compatible with Grafana versions > +8.0.0 )

If all is fine at this level, we can now focus on the different error messages related to Redis instance.

Fix getInstance error: connection refused

When you connect to Grafana, you have to define the Redis database:

• Go to Home Connections=>Data sources

• When adding all your database details and you click on “save and test” button, your can face the famous issue

For this specific case it can simply due to the fact that your Redis database is not up!

So check that in priority.

• If it is not UP, then start it and retry the “Save and test” option

• Use the redis-server command or start the Redis service using systemd on your Redis server:

nso@DBI-LT-NSO:~> redis-server

Example on Linux:

sudo systemctl start redis.service

• If you are in that specific case it should work fine now

• The second case can be because your address is not correctly set or not recognized

Below my local host name didn’t worked:

• So for those using a Docker image like me we must add the docker host IP address instead.

Get the docker host IP Address

To get the docker ip address use the following commands:

docker inspect <containerNameOrId>

Example:

C:\WINDOWS\system32>docker inspect 

8c6c36853ab0490ecca607a6aa28bd158f8f798fd221c24f973faf33be7dfb23

You can then pick the Docker ip address

Here in my example it is 172.17.0.2

“Usually, the default docker ip range is 172.17.0.0/16.

Your host should be 172.17.0.1 and your first container should be 172.17.0.2 if everything is normal and you didn’t specify any special network options.”

• Let’s try now:

• If it is still not working with Docker IP address you picked by using the docker inspect command, use the following syntax:

host.docker.internal: <your_Redis_port>

Example:

host.docker.internal: 6379

It should work now!

Fix getInstance error : ERR AUTH

This “error” message will display when you have not configured the default password user.

• So you know what to do as the error message is explicit ( check your configuration )

• Anyway, If you let the password part empty (click reset and remove the password) you can test with success

Fix getInstance error : DENIED

• This specific message is more verbose,it propose you solution to fix it such as:

a)disable the protected mode by many ways

b)setup a bind address or authentication by password

• Here are the different ways to disable the protected mode:

“DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions:

1) Just disable protected mode sending the command “CONFIG SET protected-mode no” from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent.

2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to ‘no’, and then restarting the server.

Go to your redis.cfg file and update the below line and set it to “no”

3) If you started the server manually just for testing, restart it with the ‘–protected-mode no‘ option.

4) Setup a bind address or an authentication password.

Of course : You only need to do one of the above things in order for the server to start accepting connections from the outside.

From Redis site

Example:

• For our case we will disable protected mode as it is only for test purpose.

• Use the below command on your Redis instance:

• Now you can use the save and test option it should work

Conclusion

Now you know how to fix the main issues when defining your Redis instance in Grafana.

If you are still stuck, you can also check on Grafana official site database you can find some useful info.

Feel free to check my other posts and also the dbi bloggers.

Next time we will see how to use Grafana with MongoDB.

Stay tuned!

L’article Redis: Redis X Grafana fix the “get instance error” message est apparu en premier sur dbi Blog.

Monitoring our Redis database is one of the first thing we want to do : Checking the database status , the queries consumption and also make a report on it.

For that, we will use Grafana:

It is an open source analytics & monitoring solution for every database.

Grafana can add some plugins to work on a specific database.I guess you have understood that the today’s topic is to add a Redis plugin to Grafana in order to have some dashboard and graphs.

What is Grafana?

Grafana is a free software application licensed under the GNU Affero General Public License Version 32 (formerly under the Apache 2.0 license until April 2021)

It enables data visualisation.

It can be used to create dashboards and graphs from a number of sources, including temporal databases such as Graphite, InfluxDB and OpenTSDB.

How to install Grafana

You have many ways to install this tool and it is supported on many operating systems:

Below you will have the list of the operating systems ( I will probably make another post to go deeper in details )

• Debian or Ubuntu
• RHEL or Fedora
• SUSE or openSUSE
• Grafana Docker image
• Grafana on Kubernetes
• macOS
• Windows

As we have many installation choices, we will review that later in a incoming Blog.For this post I have chosen the Docker images method.

Run Grafana Docker image

This installation will be made in two steps:

• We will see how to install the Grafana image on Docker

• We will see how to add our famous Redis plugin ( it is possible to run directly the Grafana image containing also the Redis plugin! )

Grafana docker image

Grafana Docker images come in two editions:

• Grafana Enterprise: grafana/grafana-enterprise
• Grafana Open Source: grafana/grafana-oss

For my case I have chosen the Grafana Open source Image

Docker useful commands for your grafana container

• Run Grafana via Docker CLI

To run the latest stable version of Grafana, run the following command:

docker run -d -p 3000:3000 --name=grafana grafana/grafana-enterprise

docker run -d -p 3000:3000 --name=grafana grafana/grafana-oss

• Stop the Grafana container

Use the docker ps command to show the processes running in Docker:

C:\Users\nso>docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e5fede37b3da grafana/grafana-oss "/run.sh" 8 seconds ago Up 8 seconds 0.0.0.0:3000->3000/tcp grafana

To stop the container use the command docker stop:

• docker stop CONTAINER-ID or use
• docker stop NAME, here it is grafana as previously defined

C:\Users\nso>docker stop grafana

grafana

Additional docker recommended actions

Additional actions are recommended ( we will not describe it now but it can be a topic for next time ) such as:

• Save your Grafana data
• Use Docker volumes
• Use bind mounts
• Use environment variables to configure Grafana

Run Grafana image with the Redis plugin

Install plugins in the Docker container

“You can install plugins in Grafana from the official and community plugins page or by using a custom URL to install a private plugin.

These plugins allow you to add new visualization types, data sources, and applications to help you better visualize your data.”

Grafana currently supports three types of plugins: panel, data source, and app.

For more information on managing plugins, refer to Plugin Management.

Extract from grafana official site

About Redis plugin

The Redis Data Source for Grafana allows users to connect to any Redis database On-Premises and in the Cloud.

It provides out-of-the-box predefined dashboards and lets you build customized dashboards to monitor Redis and application data.

Redis plugin requirements

• Grafana 8.0+ is required for Redis Data Source 2.X.
• Grafana 7.1+ is required for Redis Data Source 1.X.

Redis Application plugin

You can add as many data sources as you want to support multiple Redis databases. Redis Application plugin helps manage various Redis Data Sources and provides Custom panels.

Redis Explorer plugin

The Redis Explorer plugin connects to Redis Enterprise software clusters using REST API. It provides application pages to add Redis Data Sources for managed databases and dashboards to see cluster configuration.

Run the Redis plugin

Use the following command to run the Grafana image with the Redis plugins inside:

C:\Users\nso>docker run -d -p 3000:3000 --name=grafana -e "GF_INSTALL_PLUGINS=redis-datasource" grafana/grafana-oss

Here we have added the plugin name ” redis-datasource” to the image “grafana-oss”

Open Grafana

Now you have launched the Garfana image, we will check if the plugin is here:

• Open an url to your address using port 3000

• Enter the default user and password ( admin , admin )

• When done it will be asked you to update the password for security matters

Connect your database

• On Grafana select home/connections/data sources

Configure your Redis plugin

• Go to home/administration/plugins

We see that some plugins are installed by default and other can be added , here Redis plugin installed and signed.We will open it and see what it is possible to do with it.

• Open Redis plugin

• Add your data source

• You have the choice to explore or to build a new dashboard, let’s try it
• you have now the choice to add a visualization ( you are also able to import dashboard or add a library panel )
• Go to Home/Dashboards/ New dashboard and select “Add visualization”

• Now we can chose how to display our results

• Select the display for example time series, a table or a Bar chart

In the command part, you can use the Redis commands to interact with your database and of course the related keys.

• Another example here with another display with a Gauge:

Save your dashboard

• Once your dashboard is ready you click on apply and save

• Give a name to your dashboard and save it

Load your dashboard

You can now load your presets and see your saved dashboards

• Go in home/dashboards

That’s is you know how to create your own dashboard!

Conclusion

Now you know how to use Grafana to work with your Redis database.You can download the plugin by some other ways, we can see that another time if you need it.

I hope you enjoyed this blog and you have tested it on your own side.

I will prepare a troubleshooting post for this Grafana plugin installation in order to avoid you some issue I have faced

Stay tuned for next posts, you can also check my previous ones and also the dbi bloggers.

And of course don’t hesitate to take a look on the Grafana official site which is extremely well documented !

L’article Redis:Connect a Redis plugin to Grafana est apparu en premier sur dbi Blog.

Redis allows you to use many commands to create and administrate your database.

There is especially one useful feature which which allows you to execute a series of commands as a single unit of work:

The Redis transactions!

In this post will check how it works how it can be useful for your daily work.

How do transactions work?

Transactions allows to send a block of commands, using the the MULTI command.

All the commands will be queued the when using the EXEC command it will be executed.

Note:

A request sent by another client will never be served in the middle of the execution of a Redis Transaction. “This guarantees that the commands are executed as a single isolated operation“

Here is a quick example :

In a row, we created a key “PLAYERS” with the player name and their associated number, then we wanted it to be displayed using the command.

The EXEC command will commit all these actions.

Note:

When using the MULTI command, the inputs are not executed until the EXEC command is done.

What are the benefits of Redis transactions?

First one is that transactions are atomic. In other words all of the commands in the transaction are completed or none of them are.

This is an essential point for ensuring the consistency of your data, so that you don’t end up in a situation where certain commands in the transaction have been executed, but not others.

Other good point, transactions in Redis are very fast.

Remember Redis works as an in-memory database, it’s a reason it can execute transactions very quickly.

This makes it ideal for use cases where you need to make multiple changes to your data in a short period of time.

Are rollbacks possible?

Unfortunately not, but there is a good reason:

Supporting rollbacks would have a significant impact on the simplicity and performance of Redis.

How to discard a transaction?

You can use the DISCARD command to abort your transaction.

After that all it will not execute the commands within the transaction and your data will be restored to normal.

Here is an example:

• As you can see, the NUMBERS value remained the same 123456789

Watch command

Watch main function is to monitor keys, this command will make the EXEC conditional:

Following that principle, EXEC performs the transaction on the key only if this key is not WATCHed

This includes modifications made by the client, like write commands, and by Redis itself, like expiration or eviction.

If keys were modified between when they were WATCHed and when the EXEC was received, the entire transaction will be aborted instead.

Below an example of watch command:

• 1st user watch the key then start the transaction by updating the key value:

• In the mean time, the second user modifies the key with another value:

• When the 1st user issues the EXEC command, the result is an error:

• When we see the value of NUMBERS the modification was done by the 2nd user:

• You see now how the WATCH command prevents the EXEC from running

What about Optimistic locking?

Redis is able to use a check and set method to lock the Redis transaction.

For that we will use the WATCH command.

I invite you to check the Redis site to see some use cases on how optimistic locking is working

Redis scripting

Another solution is to use Redis scripts which are also transactional.

That means that you can do the same actions as with a Redis Transaction,and usually the script will be both simpler and faster.

You can have more info about scripts by following this link

Conclusion

Now you know how to perform transactions in Redis, a useful feature to keep data consistency and in a very fast way

Feel free to check my other posts and also keep an eye on dbi bloggers for new tips and tricks.

L’article Redis: How transactions work in Redis est apparu en premier sur dbi Blog.

Whereas I wanted to perform a quick demo to show some MongoDB useful command to mass

update some fields (Increment),I had a MongoDB error: “Cannot increment with a non-numeric argument”

I guess the message is explicit:

I can’t update this field as it is not a numeric field

Identify the field type:

As I have created my document with basic commands without specifying each field type, the fields are set by default (undefined )

How to know the type of your field?

For my example I have this list of players, their name, and also their number of goals.

When I wanted to increment the “goals” field by 10 it displayed the famous issue.

Let’s check the goals field type:

For that we will use the below command:

typeof db.collection.findOne().field_name

selecao_cracks> typeof db.selecao_cracks.findOne().Lastname

Here we see that Lastname field is undefined

If we take a look we see that Nickname field is a string type (I made some tests before)

The $TYPE operator:

The $type operator accepts string aliases for the BSON types in addition to the numbers corresponding to the BSON types

Update the type of the field

Use the $set operator and the $toInt operator to update the field ( In fact in that case the $set operator will replace the field and its type, later we will see that we can use another operator name $convert )

selecao_cracks> db.selecao_cracks.updateMany( {}, [{ $set : { "goals":{$toInt:"$goals"}}} ]);

• Check if the field is updated

Increment the field to check if the command is now valid

selecao_cracks> db.selecao_cracks.updateMany( {}, { $inc : { goals : 10 } });

• Now let’s check if update is done

That’s it update worked and we don’t have the error related to the data type

The other operators

We have also other operator to change our field type such as:

$toString

As it’s name says it is to change the field to a string type

$toDate

this operator allows you to modify the field as a date type

Example:

• We check the data type:

• Update the field

ballon_d_or_winner> db.ballon_d_or_forgotten.updateOne( {"name":"Zico"}, [{ $set : { "date":{$toDate:"$date"}}} ]);

• The output shows that it worked

• We see that the format is note an ISODate format

$Convert

For converting the data type of a field in MongoDB.

More information related to $Convert on the MongoDB site

This operator is very effective and can convert a field to any data type, or it can convert a field to null if the conversion is not possible.

$convert has the following syntax:

{
$convert:
{
input: ,
to: ,
onError: , // Optional.
onNull: // Optional.
}
}

In addition to $convert, MongoDB provides the following aggregation operators as shorthand when the default “onError” and “onNull” behavior is acceptable:

• $toBool
• $toDate
• $toDecimal
• $toDouble
• $toInt
• $toLong
• $toObjectId
• $toString

Conclusion

Now you know how to update a field with the correct type especially when you need to performs tasks using integer values.

I hope it helped, next time we will go deeper with these kind of operator especially combined with aggregations as it is a huge topic with many use cases.

Don’t hesitate to check my other blogs and share with dbi bloggers.

L’article MongoDB: Identify a field type est apparu en premier sur dbi Blog.