DPR and Dbvisit Standby on ODA

Introduction

Since patch version 19.21 (current one is 19.25), Oracle Database Appliance (ODA) X7, X8 and X9 will require the use of Data Preserving Reprovisioning (DPR) to apply the patch coming from 19.20 and previous releases. Unlike traditional patching, DPR erases the system disks but keeps data on ASM/ACFS disks. Data means databases, ACFS volumes, DB homes, DB Systems and VMs metadata, vDisks for DB Systems and VMs. But it doesn’t keep your system settings: specific OS configurations, additional tools, monitoring, users and groups, aso.

The reason for not being able to use classic patching method is the mandatory OS upgrade from Linux 7 to Linux 8. It is done with a fresh setup only. Actually, it’s much easier to setup a brand new OS than applying a major patch onto an existing Linux system. Therefore, using DPR will require a longer downtime of your ODA, several hours at least, meaning that you will need to switchover your databases to another ODA with Data Guard or Dbvisit Standby.

Dbvisit Standby and Oracle Database Standard Edition

Data Guard is not available on Oracle Database Standard Edition. It means that you cannot configure a Disaster Recovery setup with the embedded binaries. But a database can still be a standby as it does not rely on the database edition. A standby database is mostly a primary that cannot be updated on its own with SQL statements. If you could compare each file between a primary and a standby, the only difference you would see is a flag in the controlfile. This flag defines the role of your database. And you can still convert a primary to a standby and vice-versa with a pair of SQL commands.

Dbvisit Standby relies on this fact and comes with the tooling for transporting archivelogs, managing the sync, switching the roles, doing the failover. Dbvisit Standby has always been the smartest solution for those needing a Disaster Recovery solution without the need for buying Enterprise Edition. Among the companies I work for, 95% of the owners of ODA plus Standard Edition are using Dbvisit Standby for at least one database. When comes the time for patching an ODA, usually once a year, primaries are switched to the standby server as it would be done with Data Guard. And until now, it brings the same flexibility and convenience as Data Guard.

Best practice for Dbvisit Standby on ODA

Classic patching does not require any additional operation when using Dbvisit Standby compared to Data Guard. You just need to change the DB home location in the Dbvisit configuration file after patching the DB home. Patching a DB home is actually moving the database to a new one.

Data Preserving Reprovisioning will have more impact on Dbvisit Standby. It will erase the system, meaning Dvisit binaries and configuration files if they were located on the OS disks.

Best practice to prevent this drawback is to setup Dbvisit Standby in a dedicated ACFS filesystem. I usually create 2 filesystems, one for the binaries, logfiles and configuration files. And another one for archivelogs. This one will be sized depending on the number of protected databases and on the activity of those databases:

su – oracle

mkdir /u01/app/dbvisit/standbymp
mkdir /u01/app/dbvisit/arc
exit

su - grid

sqlplus / as sysasm
ALTER DISKGROUP RECO ADD VOLUME dbvbin SIZE 50G;
ALTER DISKGROUP RECO ADD VOLUME dbvarc SIZE 200G;

col volume_device for a40
select volume_name, VOLUME_DEVICE from v$asm_volume where volume_name='DBVBIN' or volume_name='DBVARC';

VOLUME_NAME                    VOLUME_DEVICE
------------------------------ ----------------------------------------
DBVBIN                         /dev/asm/dbvbin-156
DBVARC                         /dev/asm/dbvarc-156

exit

su - root

/sbin/mkfs -t acfs /dev/asm/dbvbin-156
/sbin/mkfs -t acfs /dev/asm/dbvarc-156

/u01/app/19.24.0.0/grid/bin/srvctl add filesystem -d /dev/asm/dbvbin-156 -g RECO -v DBVBIN -m /u01/app/dbvisit/standbymp -u oracle
/u01/app/19.24.0.0/grid/bin/srvctl start filesystem -d /dev/asm/dbvbin-156

/u01/app/19.24.0.0/grid/bin/srvctl add filesystem -d /dev/asm/dbvarc-156 -g RECO -v DBVARC -m /u01/app/dbvisit/arc -u oracle
/u01/app/19.24.0.0/grid/bin/srvctl start filesystem -d /dev/asm/dbvarc-156

df -h /u01/app/dbvisit/standbymp /u01/app/dbvisit/arc
Filesystem          Size  Used Avail Use% Mounted on
/dev/asm/dbvbin-156  50G  1.1G   49G   1% /u01/app/dbvisit/standbymp
/dev/asm/dbvarc-156  200G  1.1G  199G   1% /u01/app/dbvisit/arc

Once done, you can do the setup and configuration onto these filesystems. They will be preserved when using DPR. The only task you will have to do again after a DPR is creating the Dbvisit service, that’s it:

/u01/app/dbvisit/standbymp/bin/dbvagentmanager service install --user oracle
/u01/app/dbvisit/standbymp/bin/dbvagentmanager service start

Problem you may encounter when using DPR with Dbvisit

I already had this problem 2 times on 2 different configurations: I’m unable to get a successful preupgradereport because of inconsistent metadata in my standby databases.

odacli describe-preupgradereport -i 917d6120-5837-4130-912a-9dc79faae324
...

__DB__
...
Validate Database Status       Success  Database 'PCPDBI' is running and is    None
                                        in 'CONFIGURED' state
Validate Database Version      Success  Version '19.20.0.0.230718' for         None
                                        database 'PCPDBI' is supported
Validate Database Datapatch    Success  Role of database 'PCPDBI' is not       None
Application Status                      'Primary'. This check is skipped.
Validate TDE wallet presence   Success  Database 'PCPDBI' is not TDE enabled.  None
                                        Skipping TDE wallet presence check.
Validate Database Home         Success  Database home location check passed    None
location                                for database PCPDBI_52
Validate Database Service      Failed   The following services [PCPDBI_RW]     These services should be stopped and
presence                                created on database 'PCPDBI' can       removed, then rerun
                                        result in a failure in 'detach-node'.  'create-preupgradereport'. After
                                                                               'restore-node -d' these services
                                                                               should be restored manually
Validate Database metadata     Failed   Internal error encountered             Please check dcs-agent logs
...

The first failure is easy to fix, just remove the service associated to database role. It’s easy to put it back after DPR:

PCPDBI
srvctl stop service -db PCPDBI_52 -service PCPDBI_RW
srvctl remove service -db PCPDBI_52 -service PCPDBI_RW

The second failure does need troubleshooting, here is an extract of the dcs-agent.log:

2024-11-14 17:43:37,414 DEBUG [Validate Database metadata : PCPDBI : JobId=a19cb6d3-0acf-4f82-95b1-0e459b7cd732] [] c.o.d.c.u.CommonsUtils: Output :
Database unique name: PCPDBI_52
Database name: PCPDBI
Oracle home: /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_30
Oracle user: oracle
Spfile: +DATA/PCPDBI_52/parameterfile/spfilePCPDBI.ora
Password ******** /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_30/dbs/orapwPCPDBI ********
Domain:
Start options: mount
Stop options: abort
Database role: PHYSICAL_STANDBY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA,
RECO
Mount point paths: /u01/app/odaorahome,
/u01/app/odaorabase0
Services:
Type: SINGLE
OSDBA group: dba
OSOPER group: dbaoper
Database instance: PCPDBI
Configured nodes: rtssgtumitoda52
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed
2024-11-14 17:43:37,414 ERROR [Validate Database metadata : PCPDBI : JobId=a19cb6d3-0acf-4f82-95b1-0e459b7cd732] [] c.o.d.a.d.DetachRpcHandler: Exception
java.lang.NullPointerException: null
	at com.oracle.dcs.agent.dpr.prechecks.DatabasePrechecks$DatabaseMetadataPrecheck.validateDatabaseDbRole(DatabasePrechecks.java:779)
	at com.oracle.dcs.agent.dpr.prechecks.DatabasePrechecks$DatabaseMetadataPrecheck.execute(DatabasePrechecks.java:701)
	at com.oracle.dcs.agent.dpr.DetachRpcHandler.executePreUpgradecheck(DetachRpcHandler.java:276)
	at sun.reflect.GeneratedMethodAccessor248.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.oracle.dcs.commons.jrpc.JsonRequestProcessor.invokeRequest(JsonRequestProcessor.java:246)
	at com.oracle.dcs.commons.jrpc.JsonRequestProcessor.process(JsonRequestProcessor.java:305)
	at com.oracle.dcs.agent.task.TaskDcsJsonRpcExt.callInternal(TaskDcsJsonRpcExt.java:106)
	at com.oracle.dcs.agent.task.TaskDcsJsonRpc.call(TaskDcsJsonRpc.java:303)
	at com.oracle.dcs.agent.task.TaskDcsJsonRpc.call(TaskDcsJsonRpc.java:79)
	at com.oracle.dcs.commons.task.TaskWrapper.call(TaskWrapper.java:118)
	at com.oracle.dcs.commons.task.TaskApi.call(TaskApi.java:63)
	at com.oracle.dcs.commons.task.TaskSequential.call(TaskSequential.java:65)
	at com.oracle.dcs.commons.task.TaskSequential.call(TaskSequential.java:36)
	at com.oracle.dcs.commons.task.TaskWrapper.call(TaskWrapper.java:118)
	at com.oracle.dcs.commons.task.TaskApi.call(TaskApi.java:63)
	at com.oracle.dcs.commons.task.TaskSequential.call(TaskSequential.java:65)
	at com.oracle.dcs.commons.task.TaskSequential.call(TaskSequential.java:36)
	at com.oracle.dcs.commons.task.TaskWrapper.call(TaskWrapper.java:118)
	at com.oracle.dcs.commons.task.TaskApi.call(TaskApi.java:63)
	at com.oracle.dcs.commons.task.TaskSequential.call(TaskSequential.java:65)
	at com.oracle.dcs.commons.task.TaskSequential.call(TaskSequential.java:36)
	at com.oracle.dcs.commons.task.TaskWrapper.call(TaskWrapper.java:118)
	at com.oracle.dcs.commons.task.TaskApi.call(TaskApi.java:63)
	at com.oracle.dcs.commons.task.TaskSequential.call(TaskSequential.java:65)
	at com.oracle.dcs.commons.task.TaskSequential.call(TaskSequential.java:36)
	at com.oracle.dcs.commons.task.TaskWrapper.call(TaskWrapper.java:118)
	at com.oracle.dcs.commons.task.TaskApi.call(TaskApi.java:63)
	at com.oracle.dcs.commons.task.TaskSequential.call(TaskSequential.java:65)
	at com.oracle.dcs.agent.task.TaskLockWrapper.call(TaskLockWrapper.java:136)
	at com.oracle.dcs.agent.task.TaskLockWrapper.call(TaskLockWrapper.java:59)
	at com.oracle.dcs.commons.task.TaskWrapper.call(TaskWrapper.java:118)
	at com.oracle.dcs.commons.task.TaskApi.call(TaskApi.java:63)
	at com.oracle.dcs.commons.task.TaskSequential.call(TaskSequential.java:65)
	at com.oracle.dcs.commons.task.TaskSequential.call(TaskSequential.java:36)
	at com.oracle.dcs.commons.task.TaskWrapper.call(TaskWrapper.java:118)
	at com.oracle.dcs.commons.task.TaskWrapper.call(TaskWrapper.java:60)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

Metadata is not OK on this database, but which metadata?

As the standby is only mounted, it cannot be metadata inside the database. I had no problem when using DPR on databases protected by Data Guard, especially on standby databases. It should be the same with Dbvisit Standby.

Inconsistent metadata in OCR would make sense, but after checking on this database, and apart from database role and startup mode, it looks similar to the primary databases. I temporarily changed the database role from STANDBY to PRIMARY:

. oraenv <<< PCPDBI
srvctl modify database -db PCPDBI -role PRIMARY
srvctl stop database -db PCPDBI_52 ; sleep 10 ; srvctl start database -db PCPDBI_52 -o mount

But it doesn’t change anything.

It should be related to metadata in the ODA registry. Let’s have a look in the MySQL database where the registry is located. Stop the DCS agent before doing anything:

systemctl stop initdcsagent

cd /opt/oracle/dcs/mysql/bin/ 
./mysql -u root --socket=/opt/oracle/dcs/mysql/log/mysqldb.sock 

use dcsagentdb;
 
select name, dbrole from db where name='PCPDBI' or name='TSTDBI';
+----------+---------+
| name     | dbrole  |
+----------+---------+
| PCPDBI   | NULL    |
| TSTDBI   | PRIMARY |
+----------+---------+
2 rows in set (0.00 sec)

For some reason, there is no database role for my standby in the ODA registry, let’s solve this problem and restart the odacli stack:

Create table db_20241115 as select * from db;
Update db set dbrole='STANDBY' from db where name='PCPDBI';
Query OK, 1 rows affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Commit;
exit;

systemctl start initdcsagent

sleep 120 ; odacli create-preupgradereport -bm 
...

This solved my problem.

Conclusion

Using DPR on your ODA running Dbvisit Standby works fine, but don’t hesitate to check and fix the role of your database in the ODA registry if you’re stuck at the preupgradereport.