Dataguard Oracle 12.2 : Support for Multiple Observers

By Mouhamadou Diaw

With Oracle 12cR2 it is now possible to configure multiple observers within a single Oracle Data Guard broker configuration.
Multiple observers provide an immediate benefit for High Availability. If one observer fails, there are additional observers that can continue to monitor the status of the configuration.
In this blog we are going to talk about this new feature.

Our configuration have 3 servers:
primaserver.localdomain with db_unique_name ORCL_SITE
standserver1.localdomain with db_unique_name ORCL_SITE1
standserver2.localdomain with db_unique_name ORCL_SITE2

With Oracle 12cR2 you can register up to three observers to monitor a single Data Guard broker configuration. Each observer is identified by a name that you supply when you issue the START OBSERVER command

Let’s start with a configuration where the Fast Start Failover is disabled

DGMGRL> show configuration;
Configuration - ORCL_DR
Protection Mode: MaxProtection
Members:
ORCL_SITE - Primary database
ORCL_SITE1 - Physical standby database
ORCL_SITE2 - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS (status updated 11 seconds ago)
DGMGRL>

And let’s start an observer on each server

DGMGRL> connect sys/root@orcl_site
Connected to "ORCL_SITE"
Connected as SYSDBA.DGMGRL> start observer prima_observer
[W000 01/05 10:46:03.89] FSFO target standby is (empty)


DGMGRL> connect sys/root@ORCL_SITE1
Connected to "ORCL_SITE1"
Connected as SYSDBA.
DGMGRL> start observer standserver1_observer
[W000 01/05 10:47:19.22] FSFO target standby is (empty)


DGMGRL> connect sys/root@orcl_site2
Connected to "ORCL_SITE2"
Connected as SYSDBA.
DGMGRL> start observer standserver2_observer
[W000 01/05 10:48:25.86] FSFO target standby is (empty)

When fast-start failover is disabled the observer does not need to coordinate fast-start failover, so all observers are equivalent and have the same functionality as shown below

DGMGRL> show observer;
Configuration - ORCL_DR
Fast-Start Failover: DISABLED
Observer "prima_observer"
Host Name: primaserver.localdomain
Last Ping to Primary: 17 seconds ago
Observer "standserver1_observer"
Host Name: standserver1.localdomain
Last Ping to Primary: 2 seconds ago
Observer "standserver2_observer"
Host Name: standserver2.localdomain
Last Ping to Primary: 25 seconds ago
DGMGRL>

Now let’s enable the Fast Start Failover

DGMGRL> enable FAST_START FAILOVER
Enabled.


DGMGRL> show configuration
Configuration - ORCL_DR
Protection Mode: MaxProtection
Members:
ORCL_SITE - Primary database
ORCL_SITE1 - (*) Physical standby database
ORCL_SITE2 - Physical standby database
Fast-Start Failover: ENABLED
Configuration Status:
SUCCESS (status updated 43 seconds ago)
DGMGRL>

Now that FSFO is enabled, we have a Master observer and Backup observers. The master observer works in the same manner that a single observer worked prior to the advent of multiple observers in Oracle Database 12c Release 2 (12.2.0.1).
Only the master observer can coordinate fast-start failover with Data Guard broker. All other registered observers are considered to be backup observers. The master observer is randomly chosen by the primary and the standby.

DGMGRL> show observer;
Configuration - ORCL_DR
Primary: ORCL_SITE
Target: ORCL_SITE1
Observer "prima_observer" - Master
Host Name: primaserver.localdomain
Last Ping to Primary: 3 seconds ago
Last Ping to Target: 0 seconds ago
Observer "standserver1_observer" - Backup
Host Name: standserver1.localdomain
Last Ping to Primary: 2 seconds ago
Last Ping to Target: 3 seconds ago
Observer "standserver2_observer" - Backup
Host Name: standserver2.localdomain
Last Ping to Primary: 2 seconds ago
Last Ping to Target: 2 seconds ago
DGMGRL>

If we try to stop the master observer, we will get an error

DGMGRL> stop observer prima_observer
Error: ORA-16878: master observer cannot be stopped
Failed.
DGMGRL>

It means that if for one reason we want to stop an observer that is the master one, we have to promote a backup observer as the new master. The command SET MASTEROBSERVER is used to manually change the master observer

DGMGRL> SET MASTEROBSERVER to standserver2_observer
Sent the proposed master observer to the data guard broker configuration.
Please run SHOW OBSERVER to see if master observer switch actually happens.
DGMGRL>

We can see that now the new master is standserver2_observer

DGMGRL> show observer;
Configuration - ORCL_DR
Primary: ORCL_SITE
Target: ORCL_SITE1
Observer "standserver2_observer" - Master
Host Name: standserver2.localdomain
Last Ping to Primary: 0 seconds ago
Last Ping to Target: 1 second ago
Observer "prima_observer" - Backup
Host Name: primaserver.localdomain
Last Ping to Primary: 0 seconds ago
Last Ping to Target: 2 seconds ago
Observer "standserver1_observer" - Backup
Host Name: standserver1.localdomain
Last Ping to Primary: 1 second ago
Last Ping to Target: 1 second ago
DGMGRL>

Now if we crash the server hosting the master observer (i.e. standserver2.localdomain), one of the backup observers will automatically be the new master.

DGMGRL> show observer;
Configuration - ORCL_DR
Primary: ORCL_SITE
Target: ORCL_SITE1
Observer "prima_observer" - Master
Host Name: primaserver.localdomain
Last Ping to Primary: 2 seconds ago
Last Ping to Target: 1 second ago
Observer "standserver1_observer" - Backup
Host Name: standserver1.localdomain
Last Ping to Primary: 2 seconds ago
Last Ping to Target: 0 seconds ago
Observer "standserver2_observer" - Backup
Host Name: standserver2.localdomain
Last Ping to Primary: 119 seconds ago
Last Ping to Target: 84 seconds ago

What happens if we try to start a fourth observer?

DGMGRL> connect sys/root@ORCL_SITE1
Connected to "ORCL_SITE1"
Connected as SYSDBA.
DGMGRL> start observer fourth_observer
Error: ORA-16647: could not start more than three observers
Failed to start the observer.
DGMGRL>

As we see above, we cannot have more than 3 observers.

One important thing about multiple observers is that we have only more availability for the observer but not for the FSFO configuration. This means that if we lose at the same time the primary database and the master observer, no fast-start failover will be initiated.

Let’s represent this by some configurations
Case1: Master observer and primary database are located in the same datacenter

Fast Start Failover will not occur if we lose Primary Datacenter

Case2: Master observer and primary database are located in different datacenters

Fast Start Failover will occur if we lose Primary datacenter

Case3: Master observer and primary database are located in different datacenters

Fast Start Failover will occur if we lose Primary datacenter

Conclusion: Support of multiple observers is a very nice feature. When using this feature, master observer and primary database should not be located in the same datacenter