By Mouhamadou Diaw

In an Oracle 20c Data Guard environment with a broker configured we can have following files called client-side broker files
-The observer configuration file : observer.ora
-The observer log file
-The observer runtime datafile: fsfo.dat
-The fast-start failover callout scripts (new feature in Oracle 20c)

Before Oracle 20c, there was no default location for these files. Starting with Oracle 20c, now we can define a default location for all these files by setting an environment variable called DG_ADMIN. This variable should point to a directory. Once defined, this directory will contain subdirectories that will store the client-side broker files.

The directory which defines the $DG_ADMIN directory must be created with required permissions.

If the directory does not exist or the has wrong permissions, the broker will store the fsfo.dat file and the observer.ora file in the current directory and the log file will be redirected in the standard output.

The default directory will contain following subdirectories

admin directory : contains the observer.ora
config_ConfigurationSimpleName : contains related to the observer and callout configuration
config_ConfigurationSimpleName/log : contains the observer logfile
config_ConfigurationSimpleName/dat : contains the observer runtime data file
config_ConfigurationSimpleName/callout : contains files related to the callout configuration

In the documentation we can find following

On Linux/Unix, the directory specified by the DG_ADMIN environment variable must have read, write, and execute permissions for the directory owner only. The subdirectories that DGMGRL creates under this directory will also have the same permissions.
On Windows, the directory specified by the DG_ADMIN environment variable must have exclusive permissions wherein it can be accessed only by the current operating system user who is running DGMGRL The subdirectories created under this directory by DGMGRL will also have the same permissions.

Let’s do some practical demonstrations for this new feature. Below the configuration I am using

192.168.2.21 oraadserver : the primary database
192.168.2.22 oraadserver2 : the standby database
192.168.2.23 oraadserver3 : the observer

The Data Guard and the broker are already configured

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
DGMGRL> show configuration
Configuration - prod20
  Protection Mode: MaxPerformance
  Members:
  PROD20_SITE1 - Primary database
    PROD20_SITE2 - Physical standby database
Fast-Start Failover:  Disabled
Configuration Status:
SUCCESS   (status updated 39 seconds ago)
DGMGRL>

Now let’s connect to the oraadserver3 and let’s start the observer. Note that we do not yet define any $DG_ADMIN variable

1
2
3
4
5
6
7
8
9
oracle@oraadserver3:/home/oracle/ [prod20 (CDB$ROOT)] pwd
/home/oracle
oracle@oraadserver3:/home/oracle/ [prod20 (CDB$ROOT)] echo $DG_ADMIN
oracle@oraadserver3:/home/oracle/ [prod20 (CDB$ROOT)] nohup dgmgrl -silent sys/*******@prod20_site1 "start observer" &
[1] 8202
oracle@oraadserver3:/home/oracle/ [prod20 (CDB$ROOT)] nohup: ignoring input and appending output to ‘nohup.out’
oracle@oraadserver3:/home/oracle/ [prod20 (CDB$ROOT)]

As no DG_ADMIN variable is defined, the client-side broker files are stored in the current directory

1
2
3
4
oracle@oraadserver3:/home/oracle/ [prod20 (CDB$ROOT)] ls -ltra fsfo.dat observer_oraadserver3.log
-rw-r--r--. 1 oracle oinstall 8336 Sep  4 13:29 fsfo.dat
-rw-r-----. 1 oracle oinstall  760 Sep  4 13:30 observer_oraadserver3.log
oracle@oraadserver3:/home/oracle/ [prod20 (CDB$ROOT)]

The command show observer will confirm that the observer is actually started on the oraadserver3

1
2
3
4
5
6
7
8
9
10
11
12
13
14
DGMGRL> show observer
Configuration - prod20
  Fast-Start Failover:     DISABLED
Observer "oraadserver3"
  Host Name:                    oraadserver3
  Last Ping to Primary:         5 seconds ago
  Log File:
  State File:
DGMGRL>

And in the log file we have this

1
2
3
4
oracle@oraadserver3:/home/oracle/ [prod20 (CDB$ROOT)] less observer_oraadserver3.log
Observer 'oraadserver3' started
[W000 2020-09-04T13:29:35.168+02:00] Observer trace level is set to USER
[W000 2020-09-04T13:29:35.172+02:00] Fast-Start Failover is disabled.

Ok now let’s stop the observer and let’s define the variable DG_ADMIN. The fsfo.dat files and the observer log file are also removed.

1
2
oracle@oraadserver3:/home/oracle/ [prod20 (CDB$ROOT)] echo $DG_ADMIN/
/u01/app/oracle/admin/prod20/broker_loc/

With the following permissions

1
2
3
oracle@oraadserver3:/home/oracle/ [prod20 (CDB$ROOT)] ls -ld /u01/app/oracle/admin/prod20/broker_loc/
drwxr-xr-x. 2 oracle oinstall 6 Sep  4 13:39 /u01/app/oracle/admin/prod20/broker_loc/
oracle@oraadserver3:/home/oracle/ [prod20 (CDB$ROOT)]

And let’s start again the observer

1
2
oracle@oraadserver3:/home/oracle/ [prod20 (CDB$ROOT)] nohup dgmgrl -silent sys/***@prod20_site1 "start observer" &
[1] 9136

We can following errors in the nohup file as the permissions on folders are wrong

1
2
3
4
5
6
7
8
9
oracle@oraadserver3:/home/oracle/ [prod20 (CDB$ROOT)] nohup: ignoring input and appending output to ‘nohup.out’
[1]+  Exit 255                nohup dgmgrl -silent sys/********@prod20_site1 "start observer"
oracle@oraadserver3:/home/oracle/ [prod20 (CDB$ROOT)] cat nohup.out
DGM-17389: The directory or file $DG_ADMIN/ should not be accessible by any user other than the owner.
Connected to "prod20_site1"
DGM-17390: The directory or file $DG_ADMIN/config_prod20/ cannot be accessed.
Failed.
oracle@oraadserver3:/home/oracle/ [prod20 (CDB$ROOT)]

Now let’s change the permissions as specified in the message

1
2
3
4
[oracle@oraadserver3 ~]$ chmod -R  700 $DG_ADMIN/
[oracle@oraadserver3 ~]$ ls -ld $DG_ADMIN/
drwx------. 2 oracle oinstall 6 Sep  4 13:39 /u01/app/oracle/admin/prod20/broker_loc/
[oracle@oraadserver3 ~]$

And let’s start the observer. Then we can see following in the nohup file

1
2
3
4
5
6
Succeeded in opening the observer file "/u01/app/oracle/admin/prod20/broker_loc/config_prod20/dat/fsfo.dat".
[W000 2020-09-04T13:50:47.465+02:00] Observer could not read the contents of the observer file.
[W000 2020-09-04T13:50:47.492+02:00] FSFO target standby is
Observer 'oraadserver3' started
The observer log file is '/u01/app/oracle/admin/prod20/broker_loc/config_prod20/log/observer_oraadserver3.log'.
oracle@oraadserver3:/home/oracle/ [prod20 (CDB$ROOT)]

indeed, the observer is now started

1
2
3
4
5
6
7
8
9
10
11
12
13
14
DGMGRL> show observer
Configuration - prod20
  Fast-Start Failover:     DISABLED
Observer "oraadserver3"
  Host Name:                    oraadserver3
  Last Ping to Primary:         6 seconds ago
  Log File:
  State File:
DGMGRL>

In the $DG_ADMIN directory we can see new subdirectories created by the broker

1
2
3
4
5
6
[oracle@oraadserver3 ~]$ cd $DG_ADMIN/
[oracle@oraadserver3 broker_loc]$ pwd
/u01/app/oracle/admin/prod20/broker_loc
[oracle@oraadserver3 broker_loc]$ ls
admin  config_prod20
[oracle@oraadserver3 broker_loc]$

We remark that the config_ConfigurationSimpleName is config_prod20 as the ConfigurationSimpleName is actually prod20 in the broker configuration (by default the name of the configuration)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
DGMGRL> show configuration verbose
Configuration - prod20
  Protection Mode: MaxPerformance
  Members:
  PROD20_SITE1 - Primary database
    PROD20_SITE2 - Physical standby database
  Properties:
    FastStartFailoverThreshold      = '30'
    OperationTimeout                = '30'
    TraceLevel                      = 'USER'
    FastStartFailoverLagLimit       = '30'
    CommunicationTimeout            = '180'
    ObserverReconnect               = '0'
    FastStartFailoverAutoReinstate  = 'TRUE'
    FastStartFailoverPmyShutdown    = 'TRUE'
    BystandersFollowRoleChange      = 'ALL'
    ObserverOverride                = 'FALSE'
    ExternalDestination1            = ''
    ExternalDestination2            = ''
    PrimaryLostWriteAction          = 'CONTINUE'
    ConfigurationWideServiceName    = 'prod20_CFG'
    ConfigurationSimpleName         = 'prod20'
Fast-Start Failover:  Disabled
Configuration Status:
SUCCESS
DGMGRL>

And in the config_prod20 directory we have following subdirectorieds

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[oracle@oraadserver3 config_prod20]$ pwd
/u01/app/oracle/admin/prod20/broker_loc/config_prod20
[oracle@oraadserver3 config_prod20]$ ls -lR
.:
total 0
drwx------. 2 oracle oinstall  6 Sep  4 13:50 callout
drwx------. 2 oracle oinstall 21 Sep  4 13:50 dat
drwx------. 2 oracle oinstall 38 Sep  4 13:50 log
./callout:
total 0
./dat:
total 12
-rw-r--r--. 1 oracle oinstall 8336 Sep  4 13:50 fsfo.dat
./log:
total 32
-rw-r-----. 1 oracle oinstall 29776 Sep  4 14:53 observer_oraadserver3.log
[oracle@oraadserver3 config_prod20]$

As said before the observer log is now in the config_prod20/log directoty and the fsfo.dat is in the config_prod20/dat