Archives des Oracle - dbi Blog

OGG-30007 : How To Register Certificates In GoldenGate ?

Julien Delattre — Tue, 14 Apr 2026 06:39:00 +0000

After working on a GoldenGate deployment recently, I felt that the OGG-30007 error would be worth writing about. This error happens when registering certificates in GoldenGate. Whether you do it from the web UI or with the REST API, this GoldenGate error is detailed as such:

Processing of the certificate PEM portion of the certificate bundle resulted in more than one (1) certificate objects.

Code: OGG-30007

Cause: The read and decode processing of the specified portion of the certificate bundle produced more than one (1) objects. More than one PEM encoded object is present in the data.

Action: Review the specified portion of the certificate bundle and correct as needed. Only a single PEM encoded object is expected.

This error occurs because GoldenGate expects exactly one PEM object per import, while a certificate chain file contains multiple certificates. In the web UI, a pop-up will alert you that something is wrong:

As mentioned, this typically happens when registering a certificate chain. For instance, you could face the issue when connecting two deployments secured with NGINX. The server presents a certificate chain including the intermediate, while the client (GoldenGate) must trust both the root and the intermediate.

But when the Certificate Authority doesn’t sign your certificate directly with the Root Certificate, but with an Intermediate Certificate, the server presents a certificate chain including the intermediate. A certificate like the following will generate an OGG-30007 error if you try to add it to the truststore:

-----BEGIN CERTIFICATE-----
(intermediate)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(root)
-----END CERTIFICATE-----

And to make sure that your connections work, you should not only add the root certificate, but also the intermediate certificate. Because of the way GoldenGate stores these certificates, two separate entries must be created in the truststore. To ease monitoring and certificate management, you can name them rootCA_ogg_target and intermediateCA_ogg_target

With this, you should have no problem connecting GoldenGate deployments ! To avoid OGG-30007, ensure that each certificate is imported separately. In practice, this means extracting the root and intermediate certificates from the chain file and registering them as individual entries in the GoldenGate truststore.

L’article OGG-30007 : How To Register Certificates In GoldenGate ? est apparu en premier sur dbi Blog.

Overcome GoldenGate 26ai Bug On Custom Profiles

Julien Delattre — Thu, 09 Apr 2026 06:56:00 +0000

In GoldenGate, processes have a few options that can be managed to fit your needs. When creating an extract, for instance, you can act on the following parameters:

autoStart.enabled: whether the process will start automatically after the Administration Server starts.
autoStart.delay: delay in seconds before starting the process.
autoRestart.enabled: whether to restart the process after it fails.
autoRestart.onSuccess: the process is only restarted if it fails.
autoRestart.delay: waiting time (in seconds) before attempting to restart a process once it fails.
autoRestart.retries: maximum number of retries before stopping restart attempts.
autoRestart.window: timeframe before GoldenGate will attempt to restart the process again.
autoRestart.disableOnFailure: if set to True, GoldenGate will disable the restart if it fails to restart within the retries/window setting. You will have to start the process manually if this happens.

In the web UI, an extract profile can be customized during the third step of an extract creation called Managed Options. Here, you can choose to use the default profile, a user-defined profile or custom settings specific to the new process.

However, a bug was introduced in GoldenGate 26ai. If you create a custom profile with the adminclient, the web UI will sometimes hang when creating any new extract or replicat. This also affects 23ai installations that are patched to 26ai, so be very careful when patching an existing GoldenGate 23ai setup containing custom profiles ! After patching, you will not be able to access and manage your profiles anymore from the web UI.

What is the bug exactly ?

To reproduce the bug, simply create an extract from the adminclient.

OGG (https://vmogg ogg_test_01) 1> info profile *
                                   Auto     Delay     Auto                Wait     Reset     Disable
Name                              Start   Seconds  Restart   Retries   Seconds   Seconds  on Failure
--------------------------------  -----  --------  -------  --------  --------  --------  ----------
Default                              No                 No
OGG (https://vmogg ogg_test_01) 2> add profile TestProfile autostart no
OGG (https://vmogg ogg_test_01) 3> info profile *
                                   Auto     Delay     Auto                Wait     Reset     Disable
Name                              Start   Seconds  Restart   Retries   Seconds   Seconds  on Failure
--------------------------------  -----  --------  -------  --------  --------  --------  ----------
Default                              No                 No
TestProfile                          No                 No

In a GoldenGate 23ai web UI, you will see the following when creating an extract, in the Managed Options step:

However, in GoldenGate 26ai, the extract creation hangs indefinitely, and no error gets reported.

Before the bug is solved by Oracle, what can you do ? The first thing you can do is modify the Default GoldenGate profile, so that all extracts with this profile are affected. However, it means that you cannot fine-tune the settings for different processes, with different needs. After all, it is the reason why you should define profiles in the first place.

Is there a workaround ?

But if you still want to overcome this bug and having working profiles on 26ai until the bug gets corrected, you have two ways of doing this:

Create profiles from the web UI. You can do this in the Managed Process Profiles tab of your deployment. Profiles created through this tab do not make the web UI hang indefinitely.
Create profiles with the REST API. As presented in this blog, you can do this in two steps :
- Creation of a new profile with the Create Configuration Value endpoint on the ogg:managedProcessSettings type.
- Creation of a new configuration isDefault set to False on the ogg:configDataDescription type.

This way, you can create or recreate your profiles until the bug is solved.

L’article Overcome GoldenGate 26ai Bug On Custom Profiles est apparu en premier sur dbi Blog.

Monitoring GoldenGate Certificates Expiration

Julien Delattre — Mon, 06 Apr 2026 07:09:00 +0000

Your GoldenGate certificates should be granted by a trusted Certificate Authority, which normally handles the monitoring of certificates. However, if you don’t want your future you, or a colleague, to spend too much time debugging which certificate should be renewed, you could monitor your certificates. In this blog, I will present a way to do this with the REST API.

Certificates types in GoldenGate

The first thing you need to know is that there are three types of certificates in a GoldenGate deployment. Each of them can be monitored separately in the REST API.

Server Certificates, which belong to the server type
Client Certificates, which belong to the client type
CA Certificates, which belong to the truststore type

Certificate monitoring from the web UI

Before presenting the REST API monitoring of certificates, let’s see what we are supposed to be looking at. From the web UI, when connected to the Service Manager, you can observe the details of your certificates in the Certificate Management tab. I give below an example for both the Service Manager and a deployment in a secure GoldenGate installation.

I create short-term certificates on purpose for the example, and we see that the UI is designed to tell the user when the certificates are close to expiration. But of course, it’s better to have some sort of monitoring do the job for us.

Certificate monitoring with the REST API

With GoldenGate REST API, it is rather easy to monitor certificate expiration. Unfortunately, you cannot ask the API for a list of certificates close to expiration, so you will have to iterate over all certificates in your monitoring script.

I will use the Python client I presented in another blog, but you can of course rebuild each call manually. Here are the methods / endpoints you should use:

List Deployments – GET /services/{version}/deployments, to retrieve the list of deployments. The Service Manager is considered as a normal deployment here, so there is no need to separate it from the other deployments.
Retrieve Available Certificate Types – GET /services/{version}/deployments/{deployment}/certificates, to retrieve the collection of certificate types. It should always be the list given earlier (client, server and truststore).
Retrieve Certificate Types – GET /services/{version}/deployments/{deployment}/certificates/{type}, to get the list of certificates that belong to a specific type inside a deployment.
Retrieve Certificate Information – GET /services/{version}/deployments/{deployment}/certificates/{type}/{certificate}/info, to retrieve the information on a specific certificate.

I share below a full monitoring script that you can use to monitor all certificates in a GoldenGate setup. Feel free to adapt it to your own monitoring tool.

#!/usr/bin/env python3
"""
Oracle GoldenGate Certificate Monitoring Script
"""

from datetime import datetime, timezone
import sys

from oggrestapi import OGGRestAPI

if __name__ == '__main__':
    exit_code = 0  # 0=OK, 1=WARNING, 2=CRITICAL, 3=UNKNOWN

    try:
        # Initialize the OGG REST API client
        ogg_client = OGGRestAPI(
            url='https://vmogg:7809',
            username='ogg',
            password='ogg',
            verify_ssl=False)

        # Retrieve the list of deployments
        deployments = ogg_client.list_deployments()
        print(f"Deployments: {[d['name'] for d in deployments]}")

        # For each deployment, retrieve the list of certificates and check their expiration dates
        for deployment in deployments:
            deployment_name = deployment['name']
            print(f"    Checking certificates for deployment: {deployment_name}")

            certificate_types = ogg_client.retrieve_available_certificate_types_deployment(
                deployment=deployment_name)
            print(f"    Certificate types for deployment {deployment_name}: {[ct['name'] for ct in certificate_types]}")
            for cert_type in certificate_types:
                cert_type_name = cert_type['name']
                print(f"        Checking certificates for type: {cert_type_name}")
                certificates = ogg_client.retrieve_certificate_types(
                    deployment=deployment_name,
                    type=cert_type_name)
                for cert in certificates:
                    cert_name = cert['name']
                    print(f"            Certificate: {cert_name}")
                    certificate_information = ogg_client.retrieve_certificate_information_deployment(
                        deployment=deployment_name,
                        type=cert_type_name,
                        certificate=cert_name)
                    expiration_date = certificate_information['certificate']['validTo']
                    print(f"                Certificate Expiry Date: {expiration_date}")
                    expire_in = datetime.strptime(expiration_date, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc) - datetime.now(timezone.utc)
                    is_expired = expire_in.total_seconds() < 0

                    if is_expired:
                        print(f"                WARNING: Certificate '{cert_name}' in deployment '{deployment_name}' of type '{cert_type_name}' has expired on {expiration_date}")
                        exit_code = max(exit_code, 2)  # CRITICAL
                    else:
                        days_left = int(expire_in.total_seconds() / 86400)
                        print(f"                Certificate '{cert_name}' in deployment '{deployment_name}' of type '{cert_type_name}' will expire in {days_left} days on {expiration_date}")
                        if days_left < 30:
                            exit_code = max(exit_code, 1)  # WARNING

    except Exception as e:
        print(f"UNKNOWN: {e}")
        sys.exit(3)

    sys.exit(exit_code)

On an installation where certificates are close to expire, the output looks like this:

Deployments: ['ServiceManager', 'ogg_test_01']
    Checking certificates for deployment: ServiceManager
    Certificate types for deployment ServiceManager: ['client', 'server', 'truststore']
        Checking certificates for type: client
        Checking certificates for type: server
            Certificate: default
                Certificate Expiry Date: 2026-03-31T05:54:15Z
                Certificate 'default' in deployment 'ServiceManager' of type 'server' will expire in 0 days on 2026-03-31T05:54:15Z
        Checking certificates for type: truststore
    Checking certificates for deployment: ogg_test_01
    Certificate types for deployment ogg_test_01: ['client', 'server', 'truststore']
        Checking certificates for type: client
            Certificate: default
                Certificate Expiry Date: 2026-03-31T05:54:15Z
                Certificate 'default' in deployment 'ogg_test_01' of type 'client' will expire in 0 days on 2026-03-31T05:54:15Z
        Checking certificates for type: server
            Certificate: default
                Certificate Expiry Date: 2026-03-31T05:54:15Z
                Certificate 'default' in deployment 'ogg_test_01' of type 'server' will expire in 0 days on 2026-03-31T05:54:15Z
        Checking certificates for type: truststore
            Certificate: XCertUser-467fd0986deb
                Certificate Expiry Date: 2026-03-31T05:54:15Z
                Certificate 'XCertUser-467fd0986deb' in deployment 'ogg_test_01' of type 'truststore' will expire in 0 days on 2026-03-31T05:54:15Z
            Certificate: installed_0
                Certificate Expiry Date: 2026-03-31T05:54:15Z
                Certificate 'installed_0' in deployment 'ogg_test_01' of type 'truststore' will expire in 0 days on 2026-03-31T05:54:15Z

L’article Monitoring GoldenGate Certificates Expiration est apparu en premier sur dbi Blog.

Change GoldenGate Default Extract Profile

Julien Delattre — Thu, 02 Apr 2026 06:31:00 +0000

As part of a GoldenGate setup automation, changing the Default profile of your extracts and replicats seems like a good start if you don’t want to deal with custom profiles or changing each individual configuration.

Unfortunately, there is no easy way to modify an existing profile from the adminclient (there is no alter profile command). The Default profile makes no exception, so you will have to use the REST API for this. In this blog, I will present two ways of doing it:

Updating the Default profile directly.
Creating a custom profile, and setting it as Default.

Using the Python client for GoldenGate I presented in another blog post, you can easily create a session connecting to your GoldenGate setup and retrieve the existing configuration. For this, we will use the following methods / endpoints:

retrieve_configuration_value to get the current configuration (GET /services/{version}/config/types/{type}/values/{value})
replace_configuration_value to update the profile (PUT /services/{version}/config/types/{type}/values/{value})

from oggrestapi import OGGRestAPI

ogg_client=OGGRestAPI(
    url="https://vmogg:7810",
    username="ogg",
    password="ogg"
)

ogg_client.retrieve_configuration_value(
    value='ogg:managedProcessSettings:Default',
    type='ogg:managedProcessSettings')

This gives us the basic configuration of all new extracts and replicats in GoldenGate. Let’s see the default values:

>>> ogg_client.retrieve_configuration_value(
    value='ogg:managedProcessSettings:Default',
    type='ogg:managedProcessSettings')
{'$schema': 'ogg:managedProcessSettings', 'autoStart': {'enabled': False, 'delay': 0}, 'autoRestart': {'enabled': False, 'onSuccess': False, 'delay': 0, 'retries': 9, 'window': 60, 'disableOnFailure': True}}

Let’s have a look at the different parameters here:

autoStart.enabled: whether the process will start automatically after the Administration Server starts.
autoStart.delay: delay in seconds before starting the process.
autoRestart.enabled: whether to restart the process after it fails.
autoRestart.onSuccess: the process is only restarted if it fails.
autoRestart.delay: waiting time (in seconds) before attempting to restart a process once it fails.
autoRestart.retries: maximum number of retries before stopping restart attempts.
autoRestart.window: timeframe before GoldenGate will attempt to restart the process again.
autoRestart.disableOnFailure: if set to True, GoldenGate will disable the restart if it fails to restart within the retries/window setting. You will have to start the process manually if this happens.

Updating the `Default` profile directly

To update the Default profile, just create your own configuration and use the following example (based on the description given above) to push it to your GoldenGate deployment. Here, for instance, autoStart will be delayed by 30 seconds, and the process will try to restart every 5 minutes, and stop trying to restart after six retries. It will attempt to start again after two hours.

ogg_client.replace_configuration_value(
    value='ogg:managedProcessSettings:Default',
    type='ogg:managedProcessSettings',
    data={
        '$schema': 'ogg:managedProcessSettings',
        'autoStart': {
            'enabled': True,
            'delay': 30
        },
        'autoRestart': {
            'enabled': True,
            'retries': 6,
            'delay': 300,
            'window': 7200,
            'onSuccess': False,
            'disableOnFailure': False
        }
    }
)

We can check by retrieving the configuration again.

# Checking new configuration with the REST API
ogg_client.retrieve_configuration_value(
    value='ogg:managedProcessSettings:Default',
    type='ogg:managedProcessSettings')
{'$schema': 'ogg:managedProcessSettings', 'autoStart': {'enabled': True, 'delay': 30}, 'autoRestart': {'enabled': True, 'retries': 6, 'delay': 300, 'window': 7200, 'onSuccess': False, 'disableOnFailure': False}}

Or you can check in the web UI in the Managed Process Profiles tab.

Setting a custom profile to `Default`

If for some reason you would rather keep the Default profile and have a custom profile as default, you have to create a new profile first and set it as default. To do this, we use the create_configuration_value method / endpoint, which is the same endpoint as before but with the POST verb. If we keep the same profile as in the previous example, here is the script to run. Only the method changes, as well as the value, where Default is changed with the name of your profile (NewDefaultProfile, here).

ogg_client.create_configuration_value(
    value='ogg:managedProcessSettings:NewDefaultProfile',
    type='ogg:managedProcessSettings',
    data={
        '$schema': 'ogg:managedProcessSettings',
        'autoStart': {
            'enabled': True,
            'delay': 30
        },
        'autoRestart': {
            'enabled': True,
            'retries': 6,
            'delay': 300,
            'window': 7200,
            'onSuccess': False,
            'disableOnFailure': False
        }
    }
)

After this, you need to do two things:

Create the isDefault flag for your new profile, and set it to True. This is done with the same create_configuration_value method, but on a different type called ogg:configDataDescription.
Update the isDefault flag for the Default profile to False. Since the property already exists, we will use the replace_configuration_value method.

Here is how to do the creation of the flag:

ogg_client.create_configuration_value(
    value='ogg:managedProcessSettings:NewDefaultProfile',
    type='ogg:configDataDescription',
    data={
        'isDefault': True
    }
)

And to update the Default profile:

ogg_client.replace_configuration_value(
    value='ogg:managedProcessSettings:Default',
    type='ogg:configDataDescription',
    data={
        'isDefault': False
    }
)

From now on, any new extract or replicat will be assigned to this NewDefaultProfile !

L’article Change GoldenGate Default Extract Profile est apparu en premier sur dbi Blog.

MySQL Metadata Store service not able to start on an ODA

Marc Wagner — Tue, 31 Mar 2026 20:32:30 +0000

I was recently doing some consulting on some ODA at one of our customers, and I faced an issue with the DCS Agent and the MySQL Metadata Store. In this blog, I will show you the problem and how I could resolved it.

Problem description

On an Oracle Database Appliance, the DCS agent is the software providing the full automation to manage the appliance. It will have no impact on the well working of the databases but is mandatory to manage the appliance (patching, create dbhome, create database and any appliance resources). The DCS agent will perform various operations on the appliance, such as provisioning, database lifecycle management, backup and recovery, and storage management. The DCS Agent will use a MySQL database to store the metadata. In order for the DCS Agent to work, the MySQL database needs to be up and running.

On the node 0 of an ODA X9-2-HA, where I was preparing the next database patching operation, I wanted to run some cleanup.

[root@ODA-HA_node0 ~]# odacli cleanup-patchrepo -comp DB,GI -v 19.20.0.0.0
DCS-10009:Failed to create new service job report.

For which I could get a DCS failure.

I checked the job and realised that an ODA cleaning job where still in running status for about 3 months…

[root@ODA-HA_node0 ~]# odacli list-jobs

ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------------
80f2bfbc-a02e-47bb-b57c-754d3d6d3d3a     Server Patching                                                             2025-11-12 08:00:14 CET             Success
2e36d676-f175-402d-a7e4-735b4eb965de     Patch pre-checks for [STORAGE]                                              2025-11-12 08:58:19 CET             Success
...
c4611880-fdc8-44fd-90a5-c2dd27d12f7c     Auto purge job data                                                         2025-12-09 03:59:48 CET             Success
ed09f270-5eff-4141-9b7b-51a2e5d30e32     Auto purge job data                                                         2025-12-10 03:59:50 CET             Success
aecd8ca5-44fb-4e2c-af2b-0e42e954381a     Auto purge job data                                                         2025-12-11 03:59:52 CET             Running

[root@ODA-HA_node0 ~]#

Troubleshooting

I restarted the DCS agent.

[root@ODA-HA_node0 ~]# systemctl stop initdcsagent

[root@ODA-HA_node0 ~]# systemctl start initdcsagent

[root@ODA-HA_node0 ~]# systemctl status initdcsagent
● initdcsagent.service - Oracle dcs-agent startup
   Loaded: loaded (/etc/systemd/system/initdcsagent.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2026-03-18 14:35:41 CET; 2s ago
 Main PID: 92038 (sh)
    Tasks: 96 (limit: 79998)
   Memory: 331.7M
   CGroup: /system.slice/initdcsagent.service
           ├─92038 /bin/sh -c . /opt/oracle/dcs/bin/setupJreAgent.sh;LD_LIBRARY_PATH=/opt/oracle/rhp/lib:$JRE_HOME/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib $JAVA -Xms128m -Xmx256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC -XX:ParallelGCThreads=4 -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/opt/oracle/dcs/log/gc-dcs-agent-%t-%p.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFi>
           └─92039 /opt/oracle/dcs/java/1.8.0_441/bin/java -Xms128m -Xmx256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC -XX:ParallelGCThreads=4 -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/opt/oracle/dcs/log/gc-dcs-agent-%t-%p.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Doracle.security.jps.config=/opt/oracle/dcs/agent/jps-config.xml -Dio.nett>

Mar 18 14:35:41 ODA-HA_node0 systemd[1]: Started Oracle dcs-agent startup.
[root@ODA-HA_node0 ~]# odacli list-jobs | tail -n3

Which did not help. In the DCS Agent log I could find some errors with HAMI. Oracle HAMI is the Oracle High Availability Metadata Infrastructure service providing distributed services required by DCS including locking and synchronizing configuration details in the cluster.

[root@ODA-HA_node0 log]# view dcs-agent-infra.log
...
...
...
2026-03-18 14:59:37,776 DEBUG [DcsInfraProvisioningThread] [] c.o.d.i.h.c.CommandExecutor: Going to execute command '/opt/oracle/dcs/hami/bin/hamictl.sh ssh-equiv --ensemble ODA_DCS --hosts ODA-HA_node0-priv,ODA-HA_node1-priv --json'
2026-03-18 14:59:38,014 DEBUG [DcsInfraProvisioningThread] [] c.o.d.i.h.c.CommandExecutor: Got result from execution of '/opt/oracle/dcs/hami/bin/hamictl.sh ssh-equiv --ensemble ODA_DCS --hosts ODA-HA_node0-priv,ODA-HA_node1-priv --json':
 - RC:     1
 - OUTPUT: /opt/oracle/dcs/oda-python3/lib/python3.12/getpass.py:91: GetPassWarning: Can not control echo on the terminal.
Warning: Password input may be echoed.
hamictl_oda_dcs's password: *********
{"operation": "ssh-equiv", "result": "FAILED", "message": "Invalid password"}

2026-03-18 14:59:38,015 ERROR [DcsInfraProvisioningThread] [] c.o.d.a.i.DcsInfraProvisioningThread: Could not provision DCS infrastructure
com.oracle.dcs.infra.hams.ctl.HamiCtl$HamiCtlException: Invalid password

I decided to restart MySQL service.

[root@ODA-HA_node0 ~]# systemctl stop oda-mysql

[root@ODA-HA_node0 ~]# systemctl start oda-mysql
Job for oda-mysql.service failed because the control process exited with error code.
See "systemctl status oda-mysql.service" and "journalctl -xe" for details.

Which failed…

Checking the logs:

[root@ODA-HA_node0 ~]# journalctl -xe
...
-- Unit oda-mysql.service has begun starting up.
Mar 31 09:23:10 ODA-HA_node0 systemd[90052]: oda-mysql.service: Failed to execute command: Permission denied
Mar 31 09:23:10 ODA-HA_node0 systemd[90052]: oda-mysql.service: Failed at step EXEC spawning /opt/oracle/dcs/mysql/bin/mysqld: Permission denied

I could see that it was complaining about permissions.

I checked the permissions on the MySQL binary:

[root@ODA-HA_node0 ~]# ls -ltrh /opt/oracle/dcs/mysql/bin/mysqld
-rwxr-x--- 1 odamysql odamysql 338M Mar 26 16:55 /opt/oracle/dcs/mysql/bin/mysqld
[root@ODA-HA_node0 ~]#

And that one was ok. The linux user starting MySQL, odamysql, was having the right permissions.

I then checked all permissions including the directories:

[root@ODA-HA_node0 log]# ls -ld /opt /opt/oracle /opt/oracle/dcs /opt/oracle/dcs/mysql /opt/oracle/dcs/mysql/bin
drwxrwx---  11 root     oinstall 4096 Nov 21 13:58 /opt
drwxr-xr-x.  9 root     root     4096 Nov 14 16:31 /opt/oracle
drwxr-xr-x. 30 root     root     4096 Nov 14 12:40 /opt/oracle/dcs
drwxr-xr-x. 13 root     root     4096 Nov 14 09:10 /opt/oracle/dcs/mysql
drwxr-x---.  2 odamysql odamysql 4096 Mar 26 16:55 /opt/oracle/dcs/mysql/bin
[root@ODA-HA_node0 log]#

And I could see that there were wrong on the /opt directory… The odamysql directory is not able to enter and read in the /opt directory, as it does not belong to the group and there is no permissions on the other users…

I checked on a valid ODA and as expected the permissions should be:

drwxr-xr-x. 11 root     root     4096 Mar 18 10:02 /opt

So this is mainly due to an human mistake…

Resolution

I first checked and could confirm the issue only affect the /opt directory itself.

[root@ODA-HA_node0 opt]# pwd
/opt

[root@ODA-HA_node0 opt]# ls -ltrh
total 48K
drwxr-xr-x.  3 root root     4.0K Sep 17  2024 lsi
drwxr-xr-x   2 root root     4.0K Jan 15  2025 oracle.ahf
drwx------.  2 root root      16K Nov 11 15:42 lost+found
drwxr-xr-x.  4 root root     4.0K Nov 11 16:32 MegaRAID
drwxr-xr-x   3 root root     4.0K Nov 12 10:20 ORCLfmap
drwxr-xr-x   5 root root     4.0K Nov 12 13:36 odabr
drwxr-xr-x.  9 root root     4.0K Nov 14 16:31 oracle
drwxrwx---  16 root oinstall 4.0K Feb 16 11:19 commvault
drwx------  11 root root     4.0K Mar 30 02:08 splunkforwarder
[root@ODA-HA_node0 opt]#

I change the permissions for other users.

[root@ODA-HA_node0 /]# chmod o+rx /opt

[root@ODA-HA_node0 /]# ls -ld /opt /opt/oracle /opt/oracle/dcs /opt/oracle/dcs/mysql /opt/oracle/dcs/mysql/bin
drwxrwxr-x  11 root     oinstall 4096 Nov 21 13:58 /opt
drwxr-xr-x.  9 root     root     4096 Nov 14 16:31 /opt/oracle
drwxr-xr-x. 30 root     root     4096 Nov 14 12:40 /opt/oracle/dcs
drwxr-xr-x. 13 root     root     4096 Nov 14 09:10 /opt/oracle/dcs/mysql
drwxr-x---.  2 odamysql odamysql 4096 Mar 26 16:55 /opt/oracle/dcs/mysql/bin
[root@ODA-HA_node0 /]#

And I could successfully start the MySQL service.

[root@ODA-HA_node0 /]# systemctl start oda-mysql

[root@ODA-HA_node0 /]# systemctl status oda-mysql
● oda-mysql.service - MySQL Server for ODA
   Loaded: loaded (/etc/systemd/system/oda-mysql.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2026-03-31 09:55:48 CEST; 6s ago
     Docs: man:mysqld(8)
           http://dev.mysql.com/doc/refman/en/using-systemd.html
 Main PID: 55837 (mysqld)
   Status: "Server is operational"
    Tasks: 33 (limit: 79998)
   Memory: 423.5M
   CGroup: /system.slice/oda-mysql.service
           └─55837 /opt/oracle/dcs/mysql/bin/mysqld --defaults-file=/opt/oracle/dcs/mysql/etc/mysqldb.cnf --skip-mysqlx

Mar 31 09:55:48 ODA-HA_node0 systemd[1]: Starting MySQL Server for ODA...
Mar 31 09:55:48 ODA-HA_node0 systemd[1]: Started MySQL Server for ODA.
[root@ODA-HA_node0 /]#

And I could successfully start the DCS Agent.

[root@ODA-HA_node0 /]# systemctl status initdcsagent
● initdcsagent.service - Oracle dcs-agent startup
   Loaded: loaded (/etc/systemd/system/initdcsagent.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Wed 2026-03-18 15:22:48 CET; 1 weeks 5 days ago
 Main PID: 84546 (code=killed, signal=TERM)

Mar 19 16:18:04 ODA-HA_node0 systemd[1]: Dependency failed for Oracle dcs-agent startup.
Mar 19 16:18:04 ODA-HA_node0 systemd[1]: initdcsagent.service: Job initdcsagent.service/start failed with result 'dependency'.
Mar 24 13:38:38 ODA-HA_node0 systemd[1]: Dependency failed for Oracle dcs-agent startup.
Mar 24 13:38:38 ODA-HA_node0 systemd[1]: initdcsagent.service: Job initdcsagent.service/start failed with result 'dependency'.
Mar 25 09:46:35 ODA-HA_node0 systemd[1]: Dependency failed for Oracle dcs-agent startup.
Mar 25 09:46:35 ODA-HA_node0 systemd[1]: initdcsagent.service: Job initdcsagent.service/start failed with result 'dependency'.
Mar 26 11:06:50 ODA-HA_node0 systemd[1]: Dependency failed for Oracle dcs-agent startup.
Mar 26 11:06:50 ODA-HA_node0 systemd[1]: initdcsagent.service: Job initdcsagent.service/start failed with result 'dependency'.
Mar 30 16:07:15 ODA-HA_node0 systemd[1]: Dependency failed for Oracle dcs-agent startup.
Mar 30 16:07:15 ODA-HA_node0 systemd[1]: initdcsagent.service: Job initdcsagent.service/start failed with result 'dependency'.

[root@ODA-HA_node0 /]# systemctl start initdcsagent

[root@ODA-HA_node0 /]# systemctl status initdcsagent
● initdcsagent.service - Oracle dcs-agent startup
   Loaded: loaded (/etc/systemd/system/initdcsagent.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2026-03-31 09:57:34 CEST; 1s ago
 Main PID: 59253 (sh)
    Tasks: 96 (limit: 79998)
   Memory: 270.0M
   CGroup: /system.slice/initdcsagent.service
           ├─59253 /bin/sh -c . /opt/oracle/dcs/bin/setupJreAgent.sh;LD_LIBRARY_PATH=/opt/oracle/rhp/lib:$JRE_HOME/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib $JAVA -Xms128m -Xmx256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC -XX:ParallelGCThreads=4 -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/opt/oracle/dcs/log/gc-dcs-agent-%t-%p.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFi>
           └─59254 /opt/oracle/dcs/java/1.8.0_441/bin/java -Xms128m -Xmx256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m -XX:+DisableExplicitGC -XX:ParallelGCThreads=4 -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/opt/oracle/dcs/log/gc-dcs-agent-%t-%p.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Doracle.security.jps.config=/opt/oracle/dcs/agent/jps-config.xml -Dio.nett>

Mar 31 09:57:34 ODA-HA_node0 systemd[1]: Started Oracle dcs-agent startup.
[root@ODA-HA_node0 /]#

And I could successfully run the cleanup:

[root@ODA-HA_node0 ~]# odacli cleanup-patchrepo -comp DB,GI -v 19.20.0.0.0
{
  "jobId" : "08b395be-ae5a-4d6f-8e5d-1c0fa232f672",
  "status" : "Created",
  "message" : "",
  "reports" : [ ],
  "createTimestamp" : "March 31, 2026 11:21:58 CEST",
  "resourceList" : [ ],
  "description" : "Cleanup patchrepos",
  "updatedTime" : "March 31, 2026 11:21:58 CEST",
  "jobType" : null,
  "cpsMetadata" : null
}

[root@ODA-HA_node0 ~]# odacli describe-job -i 08b395be-ae5a-4d6f-8e5d-1c0fa232f672

Job details
----------------------------------------------------------------
                     ID:  08b395be-ae5a-4d6f-8e5d-1c0fa232f672
            Description:  Cleanup patchrepos
                 Status:  Success
                Created:  March 31, 2026 11:21:58 CEST
                Message:

Task Name                                Node Name                 Start Time                               End Time                                 Status
---------------------------------------- ------------------------- ---------------------------------------- ---------------------------------------- ----------------
Cleanup Repository                       ODA-HA_node1                 March 31, 2026 11:21:58 CEST             March 31, 2026 11:21:59 CEST             Success
Cleanup old ASR rpm                      ODA-HA_node0                 March 31, 2026 11:21:59 CEST             March 31, 2026 11:21:59 CEST             Success

[root@ODA-HA_node0 ~]#

Finally, I have also changed the group for /opt accordingly to root group.

[root@ODA-HA_node0 /]# chgrp root /opt

[root@ODA-HA_node0 /]# ls -ld /opt /opt/oracle /opt/oracle/dcs /opt/oracle/dcs/mysql /opt/oracle/dcs/mysql/bin
drwxrwxr-x  11 root     root     4096 Nov 21 13:58 /opt
drwxr-xr-x.  9 root     root     4096 Nov 14 16:31 /opt/oracle
drwxr-xr-x. 30 root     root     4096 Nov 14 12:40 /opt/oracle/dcs
drwxr-xr-x. 13 root     root     4096 Nov 14 09:10 /opt/oracle/dcs/mysql
drwxr-x---.  2 odamysql odamysql 4096 Mar 26 16:55 /opt/oracle/dcs/mysql/bin
[root@ODA-HA_node0 /]#

To wrap up…

DCS Agent and MySQL database will have no effect on the well working of your databases, but if they are not working you are not able to manage your appliance any more. Here, wrong permissions (for sure a human mistake) had sad effects on appliance software. Solving the permissions issue resolved my problem.

L’article MySQL Metadata Store service not able to start on an ODA est apparu en premier sur dbi Blog.

The Undocumented GoldenGate ‘all’ Role You Should Know About

Julien Delattre — Tue, 31 Mar 2026 05:27:00 +0000

Let me tell you a story about how I discovered an undocumented role in GoldenGate that could break your automation. Recently, I wanted to do a massive update of the role of a GoldenGate user on multiple deployments. First of all, you should know that GoldenGate users are assigned to a role within the following list: User, Operator, Administrator and Security.

I will not dwell on the differences between them, but you just have to know something important here: you cannot change the role assigned to a GoldenGate user. If you try to edit a user, the only thing that you can change is the password.

That being said, the only possible option to change the role of a user in GoldenGate is to recreate it. In the web UI, it is rather simple. You just delete it, and recreate it with the same credentials, but a different role.

But as mentioned, I wanted to do it for multiple deployments and users, without having to do the operation manually.every time. That is why I decided to use the GoldenGate REST API to quickly iterate over users and deployments. And this is where the trouble began.

According to the GoldenGate documentation on the REST API, there are two endpoints that appear to replicate the web UI behavior :

Delete User : DELETE /services/{version}/authorizations/{role}/{user}

Create User : POST /services/{version}/authorizations/{role}/{user}

However, when calling the API with these endpoints and the Python client I presented in another blog, I had the following error after trying to recreate the user:

# Deletion works as intended
>>> ogg_client.delete_user('dbiblog','Operator')
{'$schema': 'api:standardResponse', 'links': [{'rel': 'canonical', 'href': 'https://vmogg/se/Operator', 'mediaType': 'application/json'}, {'rel': 'self', 'href': 'https://vmogg/se/Operator', 'mediaType': 'application/json'}], 'messages': []}

# Creation doesn't work
>>> ogg_client.create_user('dbiblog', 'User', data={'credential':'**'})
Exception: ERROR - https://vmogg/services/ogg_test_01/adminsrvr/v2/authorizations/Operator/dbiblog: The specified user already exists.

In the UI, the user was gone, but I couldn’t recreate it with the REST API. And even from the UI, the user couldn’t be recreated. Instead, I had the following error: OGG-12430 : The specified user already exists.

At first, I thought something was wrong with either my deployment or my client, but I found nothing wrong. So I checked the restapi.log files of my deployment to understand:

What the REST API calls were doing
What the web UI was doing.

`restapi.log` analysis

Let’s try to create the exact same user (dbiblog / Operator) from the UI, and see what passes through the API. For more insight about how to query the restapi.log file efficiently, you can read a blog I wrote on the topic. Otherwise, you can just search through the logs.

As shown below, the creation of a user from the UI only generated one call to the API (excluding GET calls) to the same endpoint used by my method.

{
  "content": {
    "credential": "** Masked **",
    "type": "Basic"
  },
  "uri": "/services/v2/authorizations/Operator/dbiblog",
  "uriTemplate": "/services/{version}/authorizations/{role}/{user}",
  "verb": "POST",
  "restapi_datetime": "2026-03-27 05:40:58.776+0000",
  "restapi_reqno": 1840,
  "restapi_service": "adminsrvr",
  "restapi_status": "INFO"
}

Now, when deleting the user from the UI, two DELETE operations go through the API.

{
  "content": null,
  "uri": "/services/v2/authorizations/Operator/dbiblog",
  "uriTemplate": "/services/{version}/authorizations/{role}/{user}",
  "verb": "DELETE",
  "restapi_datetime": "2026-03-27 05:42:20.994+0000",
  "restapi_reqno": 1926,
  "restapi_service": "adminsrvr",
  "restapi_status": "INFO"
}
{
  "content": null,
  "uri": "/services/v2/authorizations/all/dbiblog",
  "uriTemplate": "/services/{version}/authorizations/{role}/{user}",
  "verb": "DELETE",
  "restapi_datetime": "2026-03-27 05:42:21.012+0000",
  "restapi_reqno": 1927,
  "restapi_service": "adminsrvr",
  "restapi_status": "INFO"
}

On top of making a DELETE call to the dbiblog / Operator pair, the UI generates another DELETE call on the dbiblog user with an undocumented all role !

This explains why you get an error when recreating a user with the REST API. If the user was deleted by a single REST API call, it still exists with the hidden all role.

To verify this hypothesis, let’s look at the list of users with the Operator and all roles before and after the user creation. Before, we only have the default ogg user (with the Security role, not shown here).

>>> ogg_client.list_users('Operator')
[]

>>> ogg_client.list_users('all')
[{'username': 'ogg'}]

But after, the newly created user also gets the all role.

>>> ogg_client.create_user(user='dbiblog', role='Operator', data={'credential':'**'})
{'$schema': 'api:standardResponse', 'links': [{'rel': 'canonical', 'href': 'https://vmogg/ser/Operator/dbiblog', 'mediaType': 'application/json'}, {'rel': 'self', 'href': 'https://vmogg/ser/Operator/dbiblog', 'mediaType': 'application/json'}], 'messages': []}

>>> ogg_client.list_users('Operator')
[{'username': 'dbiblog'}]

>>> ogg_client.list_users('all')
[{'username': 'ogg'}, {'username': 'dbiblog'}]

And of course, when deleting the user, I was only deleting it with the role I knew about.

How to recreate a GoldenGate user with the REST API ?

The conclusion is simple: to recreate a GoldenGate user with the REST API, you need to delete the user on its current role, but also on the all role. After doing this, you can recreate the user:

# User deletion on the 'Operator' role
>>> ogg_client.delete_user(user='dbiblog', role='Operator')
{'$schema': 'api:standardResponse', 'links': [{'rel': 'canonical', 'href': 'https://vmogg/Operator', 'mediaType': 'application/json'}, {'rel': 'self', 'href': 'https://vmogg/ser/Operator', 'mediaType': 'application/json'}], 'messages': []}

# User deletion on the 'all' role
>>> ogg_client.delete_user(user='dbiblog', role='all')
{'$schema': 'api:standardResponse', 'links': [{'rel': 'canonical', 'href': 'https://vmogg/ser/All', 'mediaType': 'application/json'}, {'rel': 'self', 'href': 'https://vmogg/ser/All', 'mediaType': 'application/json'}], 'messages': []}

# User creation on the 'Operator' role
>>> ogg_client.create_user(user='dbiblog', role='Operator', data={'credential':'**'})
{'$schema': 'api:standardResponse', 'links': [{'rel': 'canonical', 'href': 'https://vmogg/ser/Operator/dbiblog', 'mediaType': 'application/json'}, {'rel': 'self', 'href': 'https://vmogg/ser/Operator/dbiblog', 'mediaType': 'application/json'}], 'messages': []}

L’article The Undocumented GoldenGate ‘all’ Role You Should Know About est apparu en premier sur dbi Blog.

Starting ODA 19.29 – Old grid version cleanup

Marc Wagner — Mon, 30 Mar 2026 17:05:27 +0000

We know that during the ODA patching, a new grid version will be installed. To avoid the /u01 file system to fill up, we need to cleanup the old version. In this blog, I will show you how you can do it and also few changes coming with version 19.29, as you might need a reboot before.

Remember that until 19.28, the patch of the server was done with odacli update-server command, and this will patch the whole server (OS, ILOM, BIOS, …) and the grid in the same step.

Starting 19.29 you will run the server and the grid patching separately. odacli update-servercomponents will patch the server (OS, ILOM, HMP, firmware for local disk and controller, …) and odacli update-gihome will patch and install the new grid version. A reboot of the node will only be performed after the servercomponents update due to the ILOM and BIOS update.

Once the patching process was successfully done, we can cleanup old grid version. In my example, I patched the ODA from version 19.28 to 19.30.

Check usage

First of all we check usage of the grid files. The new grid version is 19.30, so I expect to have file using 19.30 grid.

[root@ODA01 ~]# lsof | grep "/u01/app/19.30.0.0/" | wc -l
29545

Before any cleanup, I need to ensure the old 19.28 grid infrastructure is not used any more.

[root@ODA01 ~]# lsof | grep "/u01/app/19.28.0.0/" | wc -l
3295

Which is definitely note the case.

I check which process are still using old grid infra.

[root@ODA01 ~]# lsof | grep "/u01/app/19.28.0.0/" | grep -i oakd | wc -l
3233

[root@ODA01 ~]# lsof | grep "/u01/app/19.28.0.0/" | grep -iv oakd | wc -l
62

[root@ODA01 ~]# lsof | grep "/u01/app/19.28.0.0/" | grep -iv oakd
java      86061                            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061  2390 LegacyRun            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061  2394 LegacyRun            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061  2440 LegacyRun            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061  2545 LegacyRun            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061  2546 LegacyRun            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061  2547 LegacyRun            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061  2548 LegacyRun            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061  4274 ForkJoinP            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 16159 process              root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86062 TFAServer            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86063 ParGC\x20            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86064 ParGC\x20            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86065 ParGC\x20            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86066 VM\x20Thr            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86068 Reference            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86069 Finalizer            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86071 Signal               root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86072 Service              root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86073 C2\x20Com            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86074 C1\x20Com            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86075 Sweeper              root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86078 VM\x20Per            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86079 Common-Cl            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86145 process              root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86252 LogFlushe            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86253 DataCorru            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86296 INCompres            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86297 Cleaner-1            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86298 Checkpoin            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86396 pool-3-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86397 pool-3-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86398 pool-3-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86551 TFAIndexe            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 86983 TFAMonito            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87763 ActionHan            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87776 ClusterEv            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87777 Repositor            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87782 PeriodicC            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87786 InstanceM            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87788 Thread-13            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87790 DiskUsage            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87794 process              root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87797 pool-2-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87799 PurgeMana            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87800 TFAMaster            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87804 LegacyTel            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87805 TFAPlugin            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87806 pool-2-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87807 TFAInvent            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87808 TFADbUtlS            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87809 TFADbUtlP            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 87810 RetryUplo            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 88715 Tailing              root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 88718 Tailing              root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 88719 Tailing              root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 88720 Tailing              root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 91944 pool-6-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 98253 pool-2-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 98254 pool-2-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 98331 pool-2-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar
java      86061 98388 pool-2-th            root   26r      REG             252,10     16284857    1713748 /u01/app/19.28.0.0/grid/jlib/srvm.jar

[root@ODA01 ~]# lsof | grep "/u01/app/19.28.0.0/" | grep -i oakd | tail -n3
oakd      65702 71035 oakd                 root  118r      REG             252,10       174592    1717769 /u01/app/19.28.0.0/grid/rdbms/mesg/diaus.msb
oakd      65702 71035 oakd                 root  119r      REG             252,10       174592    1717769 /u01/app/19.28.0.0/grid/rdbms/mesg/diaus.msb
oakd      65702 71035 oakd                 root  120r      REG             252,10       174592    1717769 /u01/app/19.28.0.0/grid/rdbms/mesg/diaus.msb
[root@ODA01 ~]#

So OAK and TFA are still running on old grid version and using old grid jar library.

This comes from the fact that after the server components patches, the server is rebooted, the oak, tfa and other process will start with the existing grid version, 19.28. The grid patch will not reboot the server and not restart those processes, so they will still run with old grid infra.

This is why before any cleanup and starting 19.29 only, we need to reboot the ODA.

Reboot the DOA:

[root@ODA01 ~]# systemctl reboot
[root@ODA01 ~]#

After reboot we can see that only new grid version is used:

[root@ODA01 ~]# uptime
 16:16:17 up 3 min,  1 user,  load average: 2.06, 1.58, 0.65

[root@ODA01 ~]# lsof | grep "/u01/app/19.28.0.0/" | wc -l
0

[root@ODA01 ~]# lsof | grep "/u01/app/19.30.0.0" | wc -l
45798
[root@ODA01 ~]#

Cleanup old grid

Current space for /u1 is 6.2 GB:

[root@ODA01 ~]# df -h / /opt /u01
Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/VolGroupSys-LogVolRoot   30G  5.1G   23G  19% /
/dev/mapper/VolGroupSys-LogVolOpt    48G   26G   20G  57% /opt
/dev/mapper/VolGroupSys-LogVolU01    49G   41G  6.2G  87% /u01
[root@ODA01 ~]#

Remove old version of grid:

[root@ODA01 ~]# cd /u01/app

[root@ODA01 app]# mv 19.28.0.0 BAK_19.28.0.0

[root@ODA01 app]# lsof | grep "/u01/app/19.28.0.0" | wc -l
0

[root@ODA01 app]# lsof | grep "/u01/app/BAK_19.28.0.0" | wc -l
0

[root@ODA01 app]# rm -rf BAK_19.28.0.0
[root@ODA01 app]#

Check current space:

[root@ODA01 app]# df -h / /u01 /opt /boot
Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/VolGroupSys-LogVolRoot   30G  5.1G   23G  19% /
/dev/mapper/VolGroupSys-LogVolU01    49G   27G   20G  58% /u01
/dev/mapper/VolGroupSys-LogVolOpt    48G   26G   20G  57% /opt
/dev/md126p2                        488M  218M  235M  49% /boot
[root@ODA01 app]#

We have taken back about 14 GB.

Check oracle inventory

Checking oracle inventory:

[root@ODA01 app]# cd /u01/app/oraInventory/ContentsXML/

[root@ODA01 ContentsXML]# cat inventory.xml

You will see both grid home:

The current running one is the one having CRS option to true.

Detach old grid home from the inventory

We can now detach old grid from the inventory:

[grid@ODA01 ~]$ echo $ORACLE_SID
+ASM1

[grid@ODA01 ~]$ echo $ORACLE_HOME
/u01/app/19.30.0.0/grid

[grid@ODA01 ~]$ cd $ORACLE_HOME/oui/bin

[grid@ODA01 bin]$ ./runInstaller -detachHome -silent -local ORACLE_HOME=/u01/app/19.28.0.0/grid
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 24575 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
You can find the log of this install session at:
 /u01/app/oraInventory/logs/DetachHome2026-03-29_04-34-43PM.log
'DetachHome' was successful.

Check inventory

Checking the inventory again:

[grid@ODA01 bin]$ cd /u01/app/oraInventory/ContentsXML/

[grid@ODA01 ContentsXML]$ cat inventory.xml

We will see old grid with option REMOVE=”T”

L’article Starting ODA 19.29 – Old grid version cleanup est apparu en premier sur dbi Blog.

ODA 19.30 prepatch report failing on OS patches

Marc Wagner — Mon, 30 Mar 2026 15:03:50 +0000

I had to recently patch several ODAs from 19.28 to 19.30, and one ODA patching failed running the prepatch report which is mandatory and helpful before going to patch the ODA. For each step (server, grid or storage) a prepatch report is needed. I had to troubleshoot and I could solve the problem, and I thought it would be more than interesting to share it, expecting it might be helpful.

Problem description

After dcs-admin update, dcs components update, repository update with grid and rdbms clones, I have run a prepatch report, and decided to run it already for all components : server, grid and storage.

This is why we will see in the output following part : OS, ILOM, ORACHKSERVER, SERVER, STORAGE, RHPGI, GI.

Running the prepatch report:

[root@ODA01 ~]# /opt/oracle/dcs/bin/odacli create-prepatchreport -sc -gi -st -v 19.30.0.0.0

Job details
----------------------------------------------------------------
                     ID:  6681af21-907a-4ffc-8add-d40af11441ae
            Description:  Patch pre-checks for [OS, ILOM, ORACHKSERVER, SERVER, STORAGE, RHPGI, GI] to 19.30.0.0.0
                 Status:  Created
                Created:  March 29, 2026 10:35:01 CEST
                Message:  Use 'odacli describe-prepatchreport -i 6681af21-907a-4ffc-8add-d40af11441ae' to check details of results

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------

[root@ODA01 ~]#

Describing the report, I could see that it failed on the OS rpm verification:

[root@ODA01 ~]# odacli describe-prepatchreport -i 6681af21-907a-4ffc-8add-d40af11441ae

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  6681af21-907a-4ffc-8add-d40af11441ae
            Description:  Patch pre-checks for [OS, ILOM, ORACHKSERVER, SERVER, STORAGE, RHPGI, GI] to 19.30.0.0.0
                 Status:  FAILED
                Created:  March 29, 2026 10:35:01 AM CEST
                 Result:  One or more pre-checks failed for [OS]

Node Name
---------------
ODA01

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is patch location available     Success   Patch location is available.
Verify All OS patches           Failed    DCS-10252 - dnf dry-run execution
                                          failed: check
                                          /opt/oracle/dcs/log/jobfiles/
                                          dnfdryrunout_2026-03-29_10-35-
                                          15.0673_536.log file for more details
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate all KVM ACFS           Success   All KVM ACFS resources are running
resources are running
Validate DB System VM states    Success   All DB System VMs states are expected
Validate command execution      Success   Validated command execution

__STORAGE__
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Patch location validation       Success   Verified patch location
Patch tag validation            Success   Verified patch tag
Storage patch tag validation    Success   Verified storage patch location
Verify ASM disks status         Success   ASM disks are online
Validate command execution      Success   Validated command execution

__RHPGI__
Validate available space        Success   Validated free space under /u01
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Successfully validated minimum version
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM is online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Validate space for clones       Success   Clones volume is already created
volume
Validate command execution      Success   Validated command execution

[root@ODA01 ~]#

I checked the appropriate log, and could easily guess that the process complained about missing rhn-client-tools:

[root@ODA01 ~]# view /opt/oracle/dcs/log/jobfiles/dnfdryrunout_2026-03-29_10-35-15.0673_536.log
2026-03-29 10:35:29,977 [DNFHelper] INFO    : File path /tmp/dcsfiles/dnfoperation_2026-03-29_10-35-15.0676_536.json exists
2026-03-29 10:35:29,977 [DNFHelper] INFO    : File path /tmp/dcsfiles/dnfoperation_2026-03-29_10-35-15.0676_536.json is a regular file
2026-03-29 10:35:29,979 [dnf      ] DEBUG   : User-Agent: constructed: 'libdnf (Oracle Linux Server 8.10; server; Linux.x86_64)'
2026-03-29 10:35:29,991 [dnf      ] DEBUG   : User-Agent: constructed: 'libdnf (Oracle Linux Server 8.10; server; Linux.x86_64)'
2026-03-29 10:35:30,139 [dnf      ] DEBUG   : repo: using cache for: PrecheckOSPatchRepo
2026-03-29 10:35:30,147 [dnf      ] DEBUG   : PrecheckOSPatchRepo: using metadata from Sun 29 Mar 2026 10:35:12 AM CEST.
2026-03-29 10:35:30,148 [dnf      ] INFO    : Last metadata expiration check: 0:00:18 ago on Sun 29 Mar 2026 10:35:12 AM CEST.
2026-03-29 10:35:31,907 [DNFHelper] DEBUG   : === DNF Plugin Status ===
2026-03-29 10:35:31,907 [DNFHelper] DEBUG   : DNF plugins are enabled
2026-03-29 10:35:31,910 [DNFHelper] DEBUG   : === Debug Info ===
2026-03-29 10:35:31,910 [DNFHelper] DEBUG   : Operation RPM Metadata: {"install": ["hmpipmitool.x86_64", "rhn-setup.x86_64", "uptrack-updates-5.15.0-316.196.4.1.el8uek.x86_64.noarch", "ksplice-offline.x86_64", "ksplice.x86_64"], "exclude": ["linux-firmware-core.noarch"], "remove": ["uptrack-updates-5.15.0-310.184.5.2.el8uek.x86_64.noarch"]}
2026-03-29 10:35:31,910 [DNFHelper] DEBUG   : Installed packages: ['MegaCli-8.07.07-1.noarch', 'NetworkManager-1:1.40.16-19.0.1.el8_10.x86_64', 'NetworkManager-config-server-1:1.40.16-19.0.1.el8_10.noarch', 'NetworkManager-libnm-1:1.40.16-19.0.1.el8_10.x86_64', 'NetworkManager-team-1:1.40.16-19.0.1.el8_10.x86_64', 'NetworkManager-tui-1:1.40.16-19.0.1.el8_10.x86_64', 'VRTSnbcfg-10.2.0.1-1686143553.x86_64', 'VRTSnbclibs-10.2.0.1-1686143611.x86_64', 'VRTSnbclt-10.2.0.1-1686143729.x86_64', 'VRTSnbpck-10.2.0.1-1686143786.x86_64', 'VRTSpbx-1.17.62.5-5.x86_64', 'VRTSpddea-18.0.1.0-0023.x86_64', 'abattis-cantarell-fonts-0.0.25-6.el8.noarch', 'acl-2.2.53-3.el8.x86_64', 'adcli-0.9.2-1.el8.x86_64', 'adwaita-cursor-theme-3.28.0-3.el8.noarch', 'adwaita-icon-theme-3.28.0-3.el8.noarch', 'aide-0.16-14.el8_5.1.x86_64', 'alsa-lib-1.2.10-2.el8.x86_64', 'annobin-11.13-2.0.6.el8.x86_64', 'at-3.1.20-12.el8.x86_64', 'at-spi2-atk-2.26.2-1.el8.x86_64', 'at-spi2-core-2.28.0-1.el8.x86_64', 'atk-2.28.1-1.el8.x86_64', 'attr-2.4.48-3.el8.x86_64', 'audit-3.1.2-1.0.1.el8.x86_64', 'audit-libs-3.1.2-1.0.1.el8.x
...
...
...
2026-03-29 10:35:32,505 [DNFHelper] ERROR   : Dependency solving failed
2026-03-29 10:35:32,505 [DNFHelper] ERROR   : Error occured:
Traceback (most recent call last):
  File "/opt/oracle/oak/pkgrepos/scripts/19.30.0.0.0/dnfhelper.py", line 315, in main
    metadata_file=metadata_file
  File "/opt/oracle/oak/pkgrepos/scripts/19.30.0.0.0/dnfhelper.py", line 226, in run_combined_dnf_txn
    base.resolve()
  File "/usr/lib/python3.6/site-packages/dnf/base.py", line 906, in resolve
    raise exc
dnf.exceptions.DepsolveError:
 Problem: conflicting requests
  - nothing provides rhn-client-tools = 2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe needed by rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64 from PrecheckOSPatchRepo

Analysing problem and root cause

I first checked if the package is really missing:

[root@ODA01 ~]# rpm -qa | grep -i rhn-client-tools
[root@ODA01 ~]#

And it was definitively missing, when on other ODA it was existing:

[root@ODA02 ~]# rpm -qa | grep -i rhn-client-tools
python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
[root@ODA02 ~]#

I checked in the history of the package management if one transaction could have affected this.

[root@ODA01 ~]# dnf history | grep -i rhn-client-tools
     2 | remove -y rhn-client-tools                                    | 2024-03-12 17:27 | Removed        |   16  <
[root@ODA01 ~]#

And it was the case! Someone manually removed the package 2 years ago. And also we can see that 16 dependencies rpm have been removed as well. What a mess!

I take the opportunity here to remind that an ODA IS AN APPLIANCE, and not a normal linux server. Any yum or dnf changes will have huge impact on the ODA patching, knowing that each version is coming with its bundle patch version, and the ODA will lock the repository. If a package is needed (for example, veeam agent plugin) it is always better to prioritise an installation with a tar file which will be totally transparent for the ODA patching. Otherwise it is better to install each rpm individually, avoiding if possible yum or dnf. The worst of course is removing a package with several dependencies. dnf remove including dependencies is of course something that I would never recommend to run on an ODA.

The second question we can raise, is why now? We have been running several version from 2024 to 2026 and it seems only the 19.30 version is complaining about it.

Let’s list all the dependant packages that have been removed during this dnf transaction…

[root@ODA01 ~]# dnf history info 2
Transaction ID : 2
Begin time     : Tue 12 Mar 2024 05:27:35 PM CET
Begin rpmdb    : 1162:21b3b42bf2d4360e1ace842a41737674018ff3da
End time       : Tue 12 Mar 2024 05:27:36 PM CET (1 seconds)
End rpmdb      : 1146:a140a149ea28fdac865a8abdbe954771f896a078
User           : root 
Return-Code    : Success
Releasever     : 8
Command Line   : remove -y rhn-client-tools
Comment        :
Packages Altered:
    Removed dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch         @@System
    Removed python3-dmidecode-3.12.3-2.el8.x86_64                                          @@System
    Removed python3-dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch @@System
    Removed python3-dnf-plugin-ulninfo-0.3-2.module+el8.4.0+20142+f119a2fa.noarch          @@System
    Removed python3-hwdata-2.3.6-3.el8.noarch                                              @@System
    Removed python3-librepo-1.14.2-4.el8.x86_64                                            @@System
    Removed python3-netifaces-0.10.6-4.el8.x86_64                                          @@System
    Removed python3-newt-0.52.20-11.el8.x86_64                                             @@System
    Removed python3-rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64           @@System
    Removed python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64    @@System
    Removed python3-rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64           @@System
    Removed rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                   @@System
    Removed rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64            @@System
    Removed rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                   @@System
    Removed rhnsd-5.0.35-3.0.1.module+el8+5192+3173336a.x86_64                             @@System
    Removed usermode-1.113-2.el8.x86_64                                                    @@System
[root@ODA01 yum.repos.d]#

Of course, I do not need to mention that such action has not been performed on the other ODA from the customer field, and this is why the patching did not raise this issue on the other ODAs.

[root@ODA02 ~]# dnf history | grep -i rhn-client-tools
[root@ODA02 ~]#

Resolution

The ODA had no internet access. That was a pity as I could have easily installed the needed packages that were removed from the official oracle linux distribution package manager, using dnf. So the only way of resolution for me was to download and install each package manually. In any case it was not worth to try otherwise the patching activity could have been stopped here. Stuff work as I had to download and install about 16 RPMs.

At the beginning, I tried to find the needed rpm from the Oracle linux distribution (yum.oracle.com) directly, but I need to say that it was not that so easy to find all the rpm. Also the maintenance windows was running on, and I had to find quickly a solution. I then get used of pkgs.org project which is a kind of search engine linking to the appropriate distribution mirror to download the package. We can see the home page of each project. It was much helpful because you can easily find the package, and in the download part it refers you to the exact rpm package you need from the official documentation. I found it so helpful knowing the time constraints.

For example for rhn-setup package, we can easily find it from pkgs.org:

https://oraclelinux.pkgs.org/8/ol8-appstream-x86_64/rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm.html

In the download part you will find the exact distribution link for the source and the binary, as well as the mirror, as displayed in next table:

Type	URL
Binary Package	https://yum.oracle.com/repo/OracleLinux/OL8/appstream/x86_64/getPackage/rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm
Source Package	https://yum.oracle.com/repo/OracleLinux/OL8/appstream/x86_64/getPackageSource/rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.src.rpm
Mirror	yum.oracle.com

So the package comes from the official distribution, and you can confirm for each of your downloads.

I could install the first package which are needed for rhn-client-tools:

[root@ODA01 19.30]# rpm -ivh python3-newt-0.52.20-11.el8.x86_64.rpm
warning: python3-newt-0.52.20-11.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ad986da3: NOKEY
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:python3-newt-0.52.20-11.el8      ################################# [100%]

[root@ODA01 19.30]# rpm -ivh python3-netifaces-0.10.6-4.el8.x86_64.rpm
warning: python3-netifaces-0.10.6-4.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ad986da3: NOKEY
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:python3-netifaces-0.10.6-4.el8   ################################# [100%]

[root@ODA01 19.30]# rpm -ivh python3-hwdata-2.3.6-3.el8.noarch.rpm
warning: python3-hwdata-2.3.6-3.el8.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID ad986da3: NOKEY
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:python3-hwdata-2.3.6-3.el8       ################################# [100%]

[root@ODA01 19.30]# rpm -ivh python3-dmidecode-3.12.2-15.el8.x86_64.rpm
warning: python3-dmidecode-3.12.2-15.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ad986da3: NOKEY
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:python3-dmidecode-3.12.2-15.el8  ################################# [100%]

I then tried to install rhn-client-tools and python3-rhn-client-tools:

[root@ODA01 19.30]# rpm -ivh python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm
warning: python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ad986da3: NOKEY
error: Failed dependencies:
        rhn-client-tools = 2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe is needed by python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64

[root@ODA01 19.30]# rpm -ivh rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm
warning: rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ad986da3: NOKEY
error: Failed dependencies:
        python3-rhn-client-tools = 2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe is needed by rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
[root@ODA01 19.30]#

But it was like each one needs the other, so it was not possible to install it with rpm binary. I then used dnf to install them.

[root@ODA01 19.30]# mkdir local_repo_issue_rhn
[root@ODA01 19.30]# mv rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm ./local_repo_issue_rhn/
[root@ODA01 19.30]# mv python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm ./local_repo_issue_rhn/
[root@ODA01 19.30]# cd local_repo_issue_rhn/
[root@ODA01 local_repo_issue_rhn]# dnf install *.rpm
BKB_Common                                                                                                                                                                                                                            95 kB/s | 1.5 kB     00:00
Dependencies resolved.
=====================================================================================================================================================================================================================================================================
 Package                                                           Architecture                                    Version                                                                               Repository                                             Size
=====================================================================================================================================================================================================================================================================
Installing:
 python3-rhn-client-tools                                          x86_64                                          2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe                                           @commandline                                          110 k
 rhn-client-tools                                                  x86_64                                          2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe                                           @commandline                                          394 k

Transaction Summary
=====================================================================================================================================================================================================================================================================
Install  2 Packages

Total size: 503 k
Installed size: 2.4 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                             1/1
  Installing       : rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                         1/2
  Running scriptlet: rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                         1/2
  Installing       : python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                 2/2
  Running scriptlet: python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                 2/2
  Verifying        : python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                 1/2
  Verifying        : rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                         2/2

Installed:
  python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                           rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64

Complete!
[root@ODA01 local_repo_issue_rhn]#

Working, I had now the rhn-client-tools installed:

[root@ODA01 local_repo_issue_rhn]# rpm -qa | grep -i rhn-client
rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
[root@ODA01 local_repo_issue_rhn]#

I tried a new prepatch report:

[root@ODA01 local_repo_issue_rhn]# cd
[root@ODA01 ~]# /opt/oracle/dcs/bin/odacli create-prepatchreport -sc -gi -st -v 19.30.0.0.0

Job details
----------------------------------------------------------------
                     ID:  0af9bc46-2bfe-42a1-a6dc-bb89b9c0abb8
            Description:  Patch pre-checks for [OS, ILOM, ORACHKSERVER, SERVER, STORAGE, RHPGI, GI] to 19.30.0.0.0
                 Status:  Created
                Created:  March 29, 2026 12:57:10 CEST
                Message:  Use 'odacli describe-prepatchreport -i 0af9bc46-2bfe-42a1-a6dc-bb89b9c0abb8' to check details of results

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------

[root@ODA01 ~]#

Which failed again on the same OS patches verification step:

[root@ODA01 ~]# odacli describe-prepatchreport -i 0af9bc46-2bfe-42a1-a6dc-bb89b9c0abb8

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  0af9bc46-2bfe-42a1-a6dc-bb89b9c0abb8
            Description:  Patch pre-checks for [OS, ILOM, ORACHKSERVER, SERVER, STORAGE, RHPGI, GI] to 19.30.0.0.0
                 Status:  FAILED
                Created:  March 29, 2026 12:57:10 PM CEST
                 Result:  One or more pre-checks failed for [OS]

Node Name
---------------
ODA01

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is patch location available     Success   Patch location is available.
Verify All OS patches           Failed    DCS-10252 - dnf dry-run execution
                                          failed: check
                                          /opt/oracle/dcs/log/jobfiles/
                                          dnfdryrunout_2026-03-29_12-57-
                                          23.0066_1614.log file for more details
...
...
...

And the log showed other missing packages that was part of the dependencies removed by the manual dnf remove command.

[root@ODA01 ~]# view /opt/oracle/dcs/log/jobfiles/dnfdryrunout_2026-03-29_12-57-23.0066_1614.log
...
...
...
dnf.exceptions.DepsolveError:
 Problem: package rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64 from PrecheckOSPatchRepo requires rhnsd, but none of the providers can be installed
  - package rhnsd-5.0.35-3.0.2.module+el8.10.0+90373+b70ceaf0.x86_64 from PrecheckOSPatchRepo requires rhn-check >= 0.0.8, but none of the providers can be installed
  - conflicting requests
  - nothing provides dnf-plugin-spacewalk >= 2.4.0 needed by rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64 from PrecheckOSPatchRepo

So I installed all other dependencies rpm.

I installed usermode rpm:

[root@ODA01 19.30]# rpm -ivh usermode-1.113-2.el8.x86_64.rpm
warning: usermode-1.113-2.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ad986da3: NOKEY
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing..a
   1:usermode-1.113-2.el8             ################################# [100%]
[root@ODA01 19.30]#

And finally all others with dnf:

[root@ODA01 local_repo_issue_rhn]# ls -ltrh
total 568K
-rw-r--r--. 1 root root 33K Mar 29 13:24 rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm
-rw-r--r--. 1 root root 26K Mar 29 13:31 rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm
-rw-r--r--. 1 root root 72K Mar 29 13:31 python3-rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm
-rw-r--r--. 1 root root 52K Mar 29 13:41 rhnsd-5.0.35-3.0.1.module+el8+5192+3173336a.x86_64.rpm
-rw-r--r--. 1 root root 31K Mar 29 13:53 python3-dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch.rpm
-rw-r--r--. 1 root root 54K Mar 29 14:08 python3-librepo-1.14.2-4.el8.x86_64.rpm
-rw-r--r--. 1 root root 24K Mar 29 14:08 dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch.rpm
-rw-r--r--. 1 root root 12K Mar 29 14:31 python3-dnf-plugin-ulninfo-0.3-3.module+el8.10.0+90380+96a02ce9.noarch.rpm
-rw-r--r--. 1 root root 93K Mar 29 14:31 librepo-1.14.2-4.el8.x86_64.rpm
-rw-r--r--. 1 root root 40K Mar 29 14:43 python3-rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64.rpm

[root@ODA01 local_repo_issue_rhn]# dnf install *.rpm
BKB_Common                                                                                                                                                                                                                            37 kB/s | 1.5 kB     00:00
Dependencies resolved.
=====================================================================================================================================================================================================================================================================
 Package                                                              Architecture                                   Version                                                                              Repository                                            Size
=====================================================================================================================================================================================================================================================================
Installing:
 dnf-plugin-spacewalk                                                 noarch                                         2.8.5-11.0.3.module+el8.3.0+20070+f5719e00                                           @commandline                                          24 k
 python3-dnf-plugin-spacewalk                                         noarch                                         2.8.5-11.0.3.module+el8.3.0+20070+f5719e00                                           @commandline                                          31 k
 python3-dnf-plugin-ulninfo                                           noarch                                         0.3-3.module+el8.10.0+90380+96a02ce9                                                 @commandline                                          11 k
 python3-librepo                                                      x86_64                                         1.14.2-4.el8                                                                         @commandline                                          54 k
 python3-rhn-check                                                    x86_64                                         2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe                                          @commandline                                          39 k
 python3-rhn-setup                                                    x86_64                                         2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe                                          @commandline                                          71 k
 rhn-check                                                            x86_64                                         2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe                                          @commandline                                          26 k
 rhn-setup                                                            x86_64                                         2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe                                          @commandline                                          32 k
 rhnsd                                                                x86_64                                         5.0.35-3.0.1.module+el8+5192+3173336a                                                @commandline                                          51 k
Downgrading:
 librepo                                                              x86_64                                         1.14.2-4.el8                                                                         @commandline                                          93 k

Transaction Summary
=====================================================================================================================================================================================================================================================================
Install    9 Packages
Downgrade  1 Package

Total size: 432 k
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                             1/1
  Downgrading      : librepo-1.14.2-4.el8.x86_64                                                                                                                                                                                                                1/11
  Installing       : python3-librepo-1.14.2-4.el8.x86_64                                                                                                                                                                                                        2/11
  Running scriptlet: dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch                                                                                                                                                                     3/11
  Installing       : dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch                                                                                                                                                                     3/11
  Running scriptlet: dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch                                                                                                                                                                     3/11
  Installing       : python3-dnf-plugin-ulninfo-0.3-3.module+el8.10.0+90380+96a02ce9.noarch                                                                                                                                                                     4/11
  Installing       : python3-dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch                                                                                                                                                             5/11
  Installing       : python3-rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                       6/11
  Installing       : rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                               7/11
  Installing       : rhnsd-5.0.35-3.0.1.module+el8+5192+3173336a.x86_64                                                                                                                                                                                         8/11
  Running scriptlet: rhnsd-5.0.35-3.0.1.module+el8+5192+3173336a.x86_64                                                                                                                                                                                         8/11
  Installing       : python3-rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                       9/11
  Installing       : rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                              10/11
  Cleanup          : librepo-1.14.2-5.el8.x86_64                                                                                                                                                                                                               11/11
  Running scriptlet: librepo-1.14.2-5.el8.x86_64                                                                                                                                                                                                               11/11
  Verifying        : librepo-1.14.2-4.el8.x86_64                                                                                                                                                                                                                1/11
  Verifying        : librepo-1.14.2-5.el8.x86_64                                                                                                                                                                                                                2/11
  Verifying        : dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch                                                                                                                                                                     3/11
  Verifying        : python3-dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch                                                                                                                                                             4/11
  Verifying        : python3-dnf-plugin-ulninfo-0.3-3.module+el8.10.0+90380+96a02ce9.noarch                                                                                                                                                                     5/11
  Verifying        : python3-librepo-1.14.2-4.el8.x86_64                                                                                                                                                                                                        6/11
  Verifying        : python3-rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                       7/11
  Verifying        : python3-rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                       8/11
  Verifying        : rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                               9/11
  Verifying        : rhnsd-5.0.35-3.0.1.module+el8+5192+3173336a.x86_64                                                                                                                                                                                        10/11
  Verifying        : rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                                                                                                                                                              11/11

Downgraded:
  librepo-1.14.2-4.el8.x86_64
Installed:
  dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch              python3-dnf-plugin-spacewalk-2.8.5-11.0.3.module+el8.3.0+20070+f5719e00.noarch              python3-dnf-plugin-ulninfo-0.3-3.module+el8.10.0+90380+96a02ce9.noarch
  python3-librepo-1.14.2-4.el8.x86_64                                                 python3-rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                        python3-rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
  rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                        rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64                                rhnsd-5.0.35-3.0.1.module+el8+5192+3173336a.x86_64

Complete!
[root@ODA01 local_repo_issue_rhn]#

I checked the packages were there:

[root@ODA01 local_repo_issue_rhn]# rpm -qa | grep -i rhn
python3-rhnlib-2.8.6-8.0.2.module+el8.7.0+21027+f0093b7a.noarch
rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
python3-rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
python3-rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
rhnsd-5.0.35-3.0.1.module+el8+5192+3173336a.x86_64
python3-rhn-client-tools-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
rhn-check-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
rhn-setup-2.8.16-13.0.6.module+el8.7.0+21032+057d0dfe.x86_64
rhnlib-2.8.6-8.0.2.module+el8.7.0+21027+f0093b7a.noarch
[root@ODA01 local_repo_issue_rhn]#

And rerun a precheck report:

[root@ODA01 ~]# /opt/oracle/dcs/bin/odacli create-prepatchreport -sc -gi -st -v 19.30.0.0.0

Job details
----------------------------------------------------------------
                     ID:  3d5f5d96-f566-40c7-ad84-38c436fa3bba
            Description:  Patch pre-checks for [OS, ILOM, ORACHKSERVER, SERVER, STORAGE, RHPGI, GI] to 19.30.0.0.0
                 Status:  Created
                Created:  March 29, 2026 14:46:06 CEST
                Message:  Use 'odacli describe-prepatchreport -i 3d5f5d96-f566-40c7-ad84-38c436fa3bba' to check details of results

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------

[root@ODA01 ~]#

Which was that time successful:

[root@ODA01 ~]# odacli describe-prepatchreport -i 3d5f5d96-f566-40c7-ad84-38c436fa3bba

Prepatch Report
------------------------------------------------------------------------
                 Job ID:  3d5f5d96-f566-40c7-ad84-38c436fa3bba
            Description:  Patch pre-checks for [OS, ILOM, ORACHKSERVER, SERVER, STORAGE, RHPGI, GI] to 19.30.0.0.0
                 Status:  SUCCESS
                Created:  March 29, 2026 2:46:06 PM CEST
                 Result:  All pre-checks succeeded

Node Name
---------------
ODA01

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is patch location available     Success   Patch location is available.
Verify All OS patches           Success   No dependencies found for RPMs being
                                          removed, updated and installed. Check
                                          /opt/oracle/dcs/log/jobfiles/
                                          dnfdryrunout_2026-03-29_14-46-
                                          18.0970_2312.log file for more details
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate all KVM ACFS           Success   All KVM ACFS resources are running
resources are running
Validate DB System VM states    Success   All DB System VMs states are expected
Validate command execution      Success   Validated command execution

__STORAGE__
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Patch location validation       Success   Verified patch location
Patch tag validation            Success   Verified patch tag
Storage patch tag validation    Success   Verified storage patch location
Verify ASM disks status         Success   ASM disks are online
Validate command execution      Success   Validated command execution

__RHPGI__
Validate available space        Success   Validated free space under /u01
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Successfully validated minimum version
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.30.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM is online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Validate space for clones       Success   Clones volume is already created
volume
Validate command execution      Success   Validated command execution

[root@ODA01 ~]#

I could then successfully patch the server components part:

[root@ODA01 ~]# /opt/oracle/dcs/bin/odacli update-servercomponents -v 19.30.0.0.0
{
  "jobId" : "fd6ed1d5-1982-46da-8c8c-dfa862f6427a",
  "status" : "Created",
  "message" : "A server update job has been initiated. If a reboot is required, it will be performed as part of the update process. The final status will be provided once the update operation has been completed on all nodes.",
  "reports" : [ ],
  "createTimestamp" : "March 29, 2026 15:00:09 CEST",
  "resourceList" : [ ],
  "description" : "Server Patching to 19.30.0.0.0",
  "updatedTime" : "March 29, 2026 15:00:09 CEST",
  "jobType" : null,
  "cpsMetadata" : null,
  "ecJobId" : null
}
[root@ODA01 ~]#

[root@ODA01 ~]# odacli describe-job -i fd6ed1d5-1982-46da-8c8c-dfa862f6427a

Job details
----------------------------------------------------------------
                     ID:  fd6ed1d5-1982-46da-8c8c-dfa862f6427a
            Description:  Server Patching to 19.30.0.0.0
                 Status:  Success
                Created:  March 29, 2026 15:00:09 CEST
                Message:

Task Name                                Start Time                               End Time                                 Status
---------------------------------------- ---------------------------------------- ---------------------------------------- ----------------
Modify BM udev rules                     March 29, 2026 15:00:14 CEST             March 29, 2026 15:00:26 CEST             Success
Validate ILOM server reachable           March 29, 2026 15:00:14 CEST             March 29, 2026 15:00:14 CEST             Success
Validating GI user metadata              March 29, 2026 15:00:14 CEST             March 29, 2026 15:00:14 CEST             Success
Stop oakd                                March 29, 2026 15:00:26 CEST             March 29, 2026 15:00:30 CEST             Success
Creating local repository                March 29, 2026 15:00:30 CEST             March 29, 2026 15:00:31 CEST             Success
OSPatchBaseRepo
Updating versionlock plugin              March 29, 2026 15:00:31 CEST             March 29, 2026 15:00:34 CEST             Success
Applying OS Patches                      March 29, 2026 15:00:34 CEST             March 29, 2026 15:07:16 CEST             Success
Applying HMP Patches                     March 29, 2026 15:07:16 CEST             March 29, 2026 15:07:19 CEST             Success
Creating local repository HMPPatchRepo   March 29, 2026 15:07:16 CEST             March 29, 2026 15:07:16 CEST             Success
Patch location validation                March 29, 2026 15:07:19 CEST             March 29, 2026 15:07:19 CEST             Success
Setting SELinux mode                     March 29, 2026 15:07:19 CEST             March 29, 2026 15:07:19 CEST             Success
Oda-hw-mgmt upgrade                      March 29, 2026 15:07:20 CEST             March 29, 2026 15:07:53 CEST             Success
Installing SQLcl software                March 29, 2026 15:07:53 CEST             March 29, 2026 15:07:57 CEST             Success
OSS Patching                             March 29, 2026 15:07:53 CEST             March 29, 2026 15:07:53 CEST             Success
Applying Firmware local Disk Patches     March 29, 2026 15:07:57 CEST             March 29, 2026 15:07:59 CEST             Success
Applying Firmware local Controller Patch March 29, 2026 15:07:59 CEST             March 29, 2026 15:11:23 CEST             Success
Applying Firmware shared Controller      March 29, 2026 15:11:23 CEST             March 29, 2026 15:11:42 CEST             Success
Patch
Checking Ilom patch Version              March 29, 2026 15:11:43 CEST             March 29, 2026 15:11:43 CEST             Success
Disabling IPMI v2                        March 29, 2026 15:11:43 CEST             March 29, 2026 15:11:44 CEST             Success
Patch location validation                March 29, 2026 15:11:43 CEST             March 29, 2026 15:11:43 CEST             Success
Save password in Wallet                  March 29, 2026 15:11:43 CEST             March 29, 2026 15:11:43 CEST             Success
Apply Ilom patch                         March 29, 2026 15:11:44 CEST             March 29, 2026 15:22:19 CEST             Success
Copying Flash Bios to Temp location      March 29, 2026 15:22:19 CEST             March 29, 2026 15:22:19 CEST             Success
Start oakd                               March 29, 2026 15:22:20 CEST             March 29, 2026 15:22:36 CEST             Success
Add SYSNAME in Env                       March 29, 2026 15:22:37 CEST             March 29, 2026 15:22:37 CEST             Success
Cleanup JRE Home                         March 29, 2026 15:22:37 CEST             March 29, 2026 15:22:37 CEST             Success
Starting the clusterware                 March 29, 2026 15:22:37 CEST             March 29, 2026 15:23:29 CEST             Success
Update lvm.conf file                     March 29, 2026 15:23:30 CEST             March 29, 2026 15:23:30 CEST             Success
Generating and saving BOM                March 29, 2026 15:23:31 CEST             March 29, 2026 15:25:37 CEST             Success
Update System full patch version         March 29, 2026 15:23:31 CEST             March 29, 2026 15:23:31 CEST             Success
Update System rebootless patch version   March 29, 2026 15:23:31 CEST             March 29, 2026 15:23:31 CEST             Success
PreRebootNode Actions                    March 29, 2026 15:25:37 CEST             March 29, 2026 15:26:16 CEST             Success
Reboot Node                              March 29, 2026 15:26:16 CEST             March 29, 2026 15:39:48 CEST             Success

[root@ODA01 ~]#

To wrap up…

ODA is an appliance. Any bad behaviour with the rpm and dependencies will bring life more difficult to the consultant coming to patch your ODA. So thanks for sparing a thought for the consultant who will have to patch your ODA before you start playing with yum or dnf on your ODA.

L’article ODA 19.30 prepatch report failing on OS patches est apparu en premier sur dbi Blog.

Deployment Creation INS-85037 Error With GoldenGate 26ai for DB2 z/OS

Julien Delattre — Thu, 26 Mar 2026 07:38:00 +0000

Among all the automation I was doing around a GoldenGate installation for DB2, I recently ended up with an INS-85037 error when running the configuration assistant oggca.sh. And because this error is quite common and has many different possible root causes, I wanted to write about it.

Differences in GoldenGate setup between Oracle and DB2

If you’re wondering how to set up GoldenGate 26ai for DB2 z/OS, it is very similar to what you would do with GoldenGate for Oracle. For more information on standard GoldenGate setups, you can read my blog posts about both 26ai and 23ai installations.

For the binary installation, the main difference is that INSTALL_OPTION should be set to DB2ZOS. A complete oggcore.rsp response file would look like this:

oracle.install.responseFileVersion=/oracle/install/rspfmt_ogginstall_response_schema_v23_1_0
INSTALL_OPTION=DB2ZOS
SOFTWARE_LOCATION=/u01/app/oracle/product/oggzos
INVENTORY_LOCATION=/u01/app/oraInventory
UNIX_GROUP_NAME=oinstall

When running the configuration assistant, some options are not available, but the main difference is in the environment variables section of the response file. You should have an IBMCLIDRIVER variable set to your DB2 driver’s path.

ENV_LD_LIBRARY_PATH=${IBMCLIDRIVER}/1ib:${OGG_HOME}/1ib
IBMCLIDRIVER=/path/to/ibmclidriver
ENV_USER_VARS=

Possible Solution for INS-85037

That being said, here is the exact error I had when running the Configuration Assistant oggca.sh:

[FATAL] [INS-85037] Deployment creation failed.
  ACTION: Check logs at /u01/app/oraInventory/logs/OGGCAConfigActions2026-03-22_15-19-15PM for more information.
*MORE DETAILS*
Return code 503 (Service Unavailable) does not match the expected code 201 (Created).
Verification failed for REST call to 'http://127.0.0.1:7810/services/v2/authorizations/security/ogguser'
Results for "Add a new deployment":
..."Verifying Service Manager deployment status.": SUCCEEDED
..."Adding 'zos_test' deployment.": SUCCEEDED
...."Configuring and starting the Administration Service.": SUCCEEDED
..."Verifying the initial Administration Service configuration.": SUCCEEDED
..."Adding user 'ogguser' to administer the deployment.": FAILED
Log of this session available at: /u01/app/oraInventory/logs/OGGCAConfigActions2026-03-22_15-19-15PM
The deployment creation failed and the associated files will be deleted from disk. Oracle recommends that if you want to keep the log files, you should move them to another location.
Log files will be copied to:
/u01/app/oraInventory/logs/OGGCAConfigActions2026-03-22_15-19-15PM/userdeploy_logs_2026-03-22_15-19-15PM
[WARNING] [INS-32090] Software installation was unsuccessful.
ACTION: Refer to the log files for details or contact Oracle Support Services.

Unfortunately, the installation logs did not show anything other than the following:

SEVERE: Deployment creation job failed.
INFO: Service Manager deployment that was created as part of the process needs to be removed.
INFO: Running clean-up job for Service Manager.
SEVERE: Removing Service Manager deployment.

The deployment and the service manager get deleted after the installation failure, but the logs are also copied to the oraInventory installation logs. Looking at the ServiceManager.log in the smdeploy folder, we don’t get much information.

ERROR| Configuration does not contain a 'config/network' specification. (ServiceManager.Topology)

The same applies to the restapi.log, where the logs start after the initial deployment creation error. Unfortunately, none of this was really helpful in my case. After quite some digging, I found that the response file I was using when running oggca.sh had an error. In the custom section for environment variables, I had the following settings:

# SECTION G - ENVIRONMENT VARIABLES
ENY_LD_LIBRARY_PATH-S{IBMCLIDRIVER}/1ib:${OGG_HOME}/11b
IBMCLIDRIVER=/u01/app/ibm/db2_odbc_cli_11_5
ENV_USER_VARS=

It looks like what I gave earlier, except that the path for the clidriver was incomplete.

oracle@vmogg:/home/oracle/ [ogg] ls -l /u01/app/ibm/db2_odbc_cli_11_5
drwxr-xr-x 3 oracle oinstall 23 Mar 22 2026 odbc_cli

oracle@vmogg:/home/oracle/ [ogg] ls -l /u01/app/ibm/db2_odbc_cli_11_5/odbc_cli/clidriver/
-r-xr-xr-x 1 oracle oinstall 4170 Mar 17 2021 Readme.txt
drwxr-xr-x 2 oracle oinstall 36 Mar 22 2026 adm
drwxr-xr-x 2 oracle oinstall 122 Mar 22 2026 bin
drwxr-xr-x 2 oracle oinstall 197 Mar 22 2026 bnd
drwxr-xr-x 2 oracle oinstall 157 Mar 22 09:16 cfg
drwxr-xr-x 2 oracle oinstall 24 Mar 22 2026 cfecache
drwxr-xr-x 4 oracle oinstall 27 Mar 22 2026 conv
drwxr-xr-x 3 oracle oinstall 49 Mar 22 09:26 db2dump
drwxr-xr-x 3 oracle oinstall 217 Mar 22 2026 lib
drwxr-xr-x 3 oracle oinstall 124 Mar 22 09:26 license
drwxr-xr-x 3 oracle oinstall 28 Mar 22 2026 msg
drwxr-xr-x 3 oracle oinstall 21 Mar 22 2026 properties
drwxr-xr-x 3 oracle oinstall 20 Mar 22 2026 security32
drwxr-xr-x 3 oracle oinstall 20 Mar 22 2026 security64

After correcting the oggca.rsp response file with the correct path, the configuration assistant ran successfully.

oracle@vmogg:/u01/app/oracle/product/ogg26/bin [ogg] oggca.sh -silent -responseFile /home/oracle/oggca.rsp
Successfully Setup Software.

Next time you encounter an error like this when setting up GoldenGate for DB2, make sure to check not only the variable value but also the actual content of the IBMCLIDRIVER directory !

NB: If you had this error for any other kind of setup, make sure to always check all the content of the response file you are using, as well as the prerequisites. (CLIDRIVER in this case, but it could be XAG, etc.)

L’article Deployment Creation INS-85037 Error With GoldenGate 26ai for DB2 z/OS est apparu en premier sur dbi Blog.

Discover refreshable clone PDB with Autoupgrade

Xavier Franck Godard — Tue, 24 Mar 2026 14:12:49 +0000

AutoUpgrade with a refreshable clone is basically “zero‑panic upgrades with a live copy of your database.”.

What problem it solves

Traditionally you had to schedule a maintenance window, stop everything, take a backup, upgrade, and hope nothing went wrong.

With a refreshable clone PDB, AutoUpgrade builds and continuously syncs a copy of your database while production stays online. At cutover time, you just stop users, do a last refresh, convert/upgrade the clone, and switch them over. If something goes wrong, the original source is untouched and you can fall back quickly.

Core idea in simple terms

Think of your non‑CDB or old‑version PDB as the “master” and the refreshable clone PDB as a “follow‑me” copy sitting in the target CDB.

AutoUpgrade:

Creates a PDB in the target CDB via database link (initial clone of datafiles).
Marks it as refreshable, so redo from the source is applied and it keeps rolling forward.
Lets you test the clone (read‑only) while users are still working on the source.
At a controlled start time, runs a last refresh, disconnects it from the source, converts it to a normal PDB, and upgrades it.

From your point of view: you prepare everything days in advance, and the real downtime shrinks to “final refresh + upgrade + app switch.”

High‑level lifecycle

For a non‑CDB to PDB migration or a PDB upgrade, the flow looks like this:

Preparation

You have a source: non‑CDB 12.2/19c or older PDB.
You have a target: a higher‑version CDB (for example 23ai/26ai) with enough space and network.

Initial deploy with AutoUpgrade

You configure AutoUpgrade with the source and target, plus the parameter telling it to use refreshable clone PDB.
In deploy mode, AutoUpgrade creates the pluggable database in the target CDB via DB link, copies the datafiles, and defines it as refreshable.
From now on, redo is shipped from source to target and applied, so the clone stays close to current.

Sync phase (low‑risk testing)

The source database stays fully online; business keeps running.
The refreshable clone is read‑only, so you can query it, run app smoke tests, check performance characteristics, etc.
AutoUpgrade keeps the job running in the background, doing periodic refreshes.

Cutover (final refresh and conversion)

When you reach the maintenance window, users leave the system and you quiesce activity on the source.
AutoUpgrade performs a final refresh: last redo from source is applied on the clone so you don’t lose any committed data.
The clone is then disconnected from the source, turned into a regular PDB, and AutoUpgrade moves into the upgrade and conversion steps (non‑CDB to PDB conversion if needed, then catalog/PSU/UTLRP, etc.).

Post‑upgrade and fallback

You point applications to the new PDB in the target CDB.
The original source database still exists; if you hit a show‑stopper, you can redirect apps back to it and plan a new attempt.

In practice, the “scary” part is only the final refresh and the moment you switch your apps.

Why DBAs like this pattern

Some clear advantages:

Minimal downtime: Most of the heavy lifting (copy + sync) happens while production is running; downtime is limited to final refresh and upgrade.
Built‑in rollback: Because the source stays untouched, you always have a clean fallback without restore/recovery.
Realistic testing: You test against a clone built from real production data that is almost up‑to‑date, not a weeks‑old backup.
Automation: AutoUpgrade orchestrates the create‑clone, refresh, disconnect, convert, and upgrade steps; you mostly steer with parameters and commands instead of custom scripts.

Trade‑offs are mainly around resources: you need disk, CPU, and network to maintain the refreshable clone, and you have to ensure redo shipping is reliable (archivelog gaps or network glitches can break the refresh and need fixing).

Typical example

Imagine you need to move a 19c non‑CDB to a new 26ai CDB on a different host, with less than 30 minutes downtime:

Monday: you configure AutoUpgrade with the refreshable clone option, start the job. The tool creates the PDB clone in the 26ai CDB and starts streaming redo. Users never notice.
Next days: you let it refresh every few minutes, developers connect read‑only to the clone and test their application against 26ai. Everything looks good.
Saturday night: you enter the maintenance window, let open transactions finish, stop app traffic, and tell AutoUpgrade to proceed to the final refresh. Once that’s done, it disconnects the clone, upgrades it, and runs post‑upgrade steps.
After checks, you change the service names on the app side so they point to the new PDB. Your downtime is mostly spent waiting for the upgrade scripts, not copying terabytes of data.

L’article Discover refreshable clone PDB with Autoupgrade est apparu en premier sur dbi Blog.

Archives des Oracle - dbi Blog

OGG-30007 : How To Register Certificates In GoldenGate ?

Overcome GoldenGate 26ai Bug On Custom Profiles

What is the bug exactly ?

Is there a workaround ?

Monitoring GoldenGate Certificates Expiration

Certificates types in GoldenGate

Certificate monitoring from the web UI

Certificate monitoring with the REST API

Change GoldenGate Default Extract Profile

Updating the Default profile directly

Setting a custom profile to Default

MySQL Metadata Store service not able to start on an ODA

Problem description

Troubleshooting

Resolution

To wrap up…

The Undocumented GoldenGate ‘all’ Role You Should Know About

restapi.log analysis

How to recreate a GoldenGate user with the REST API ?

Starting ODA 19.29 – Old grid version cleanup

Check usage

Cleanup old grid

Check oracle inventory

Detach old grid home from the inventory

Check inventory

ODA 19.30 prepatch report failing on OS patches

Problem description

Analysing problem and root cause

Resolution

To wrap up…

Deployment Creation INS-85037 Error With GoldenGate 26ai for DB2 z/OS

Differences in GoldenGate setup between Oracle and DB2

Possible Solution for INS-85037

Discover refreshable clone PDB with Autoupgrade

What problem it solves

Core idea in simple terms

High‑level lifecycle

Preparation

Initial deploy with AutoUpgrade

Sync phase (low‑risk testing)

Cutover (final refresh and conversion)

Post‑upgrade and fallback

Why DBAs like this pattern

Typical example

Updating the `Default` profile directly

Setting a custom profile to `Default`

`restapi.log` analysis