By Mouhamadou Diaw

With Oracle 21c (In fact Oracle 20c), we have a nice new feature with the broker which is the Fast-start Failover Callout Configuration scripts.
I talked about this feature in some presentations I did about the new features with Oracle 21c for the broker
This feature will allow to automate tasks that must be performed before and after a fast-start failover operation.

Yes this means that with Oracle 21c, we can configure some tasks to be executed before and after doing a fast_start failover. Sure that this feature will make life easier.

The use of this feature is very simple

We have to configure the DG_ADMIN variable which will be the default location of Client-side Broker Files. See my previous blog

The callout scripts should be located in the $DG_ADMIN/config_ConfigurationSimpleName/callout.
The configuration file must be named fsfocallout.ora, and it’s mandatory.
This fsfocallout.ora file will contain the names of the pre-callout configuration script and post-callout configuration script.

The pre_callout script must create a .suc file and a .err file. Before doing a FSFO, then the observer will check for .suc file or a for a .err file. If a .suc file exists then the observer will initiate a FSFO.

The .suc file and the .err file must be at the same location that the callout scripts.

Below the configuration I am using

oraadserver : 192.168.2.21 for the primary database
oraadserver2 : 192.168.2.22 for the standby database
oraadserver3 : 192.168.2.23 for the observer

The FSFO is already configured

1
2
3
4
5
6
7
8
9
10
11
12
13
DGMGRL> show configuration
Configuration - db21
  Protection Mode: MaxPerformance
  Members:
  DB21_SITE1 - Primary database
    DB21_SITE2 - (*) Physical standby database
Fast-Start Failover: Enabled in Potential Data Loss Mode
Configuration Status:
SUCCESS   (status updated 5 seconds ago)

And the observer is running

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
DGMGRL> show observer
Configuration - db21
  Fast-Start Failover:     ENABLED
  Primary:            DB21_SITE1
  Active Target:      DB21_SITE2
Observer "oraadserver3" - Master
  Host Name:                    oraadserver3
  Last Ping to Primary:         2 seconds ago
  Last Ping to Target:          0 seconds ago
  Log File:                     /u01/app/oracle/admin/prod20/broker_files/config_db21/log/observer_oraadserver3.log
  State File:                   /u01/app/oracle/admin/prod20/broker_files/config_db21/dat/fsfo.dat
DGMGRL>

On my observer server, the $DG_ADMIN variable is configured (directories of the DG_ADMIN must have some prerequisites (see previous blog or the documentation )

1
2
3
[oracle@oraadserver3 callout]$ echo $DG_ADMIN/
/u01/app/oracle/admin/prod20/broker_files/
[oracle@oraadserver3 callout]$

And in the $DG_ADMIN

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
[oracle@oraadserver3 callout]$ ls -lR $DG_ADMIN/
/u01/app/oracle/admin/prod20/broker_files/:
total 0
drwx------. 2 oracle oinstall  6 Feb  3  2021 admin
drwx------. 5 oracle oinstall 40 Feb 25 11:35 config_db21
/u01/app/oracle/admin/prod20/broker_files/admin:
total 0
/u01/app/oracle/admin/prod20/broker_files/config_db21:
total 4
drwx------. 2 oracle oinstall   99 Mar  4 16:36 callout
drwx------. 2 oracle oinstall   21 Nov 29 13:50 dat
drwx------. 2 oracle oinstall 4096 Feb 25 13:35 log
/u01/app/oracle/admin/prod20/broker_files/config_db21/callout:
total 12
-rwx------. 1 oracle oinstall 979 Mar  4 16:24 fsfocallout.ora
-rwx------. 1 oracle oinstall  57 Feb 25 11:36 fsfo_postcallout
-rwx------. 1 oracle oinstall 351 Mar  4 16:35 fsfo_precallout
-rw-r--r--. 1 oracle oinstall   0 Mar  4 16:36 fsfo_precallout.suc
/u01/app/oracle/admin/prod20/broker_files/config_db21/dat:
total 12
-rw-r--r--. 1 oracle oinstall 8344 Mar  4 16:48 fsfo.dat
/u01/app/oracle/admin/prod20/broker_files/config_db21/log:
total 84
-rw-r-----. 1 oracle oinstall 66093 Mar  4 16:48 observer_oraadserver3.log
[oracle@oraadserver3 callout]$

In the callout directory, we have the FSFO callout scripts

1
2
3
4
5
6
7
8
oracle@oraadserver:/u01/app/oracle/admin/prod20/broker_files/config_db21/callout/ [DB21 (CDB$ROOT)] pwd
/u01/app/oracle/admin/prod20/broker_files/config_db21/callout
oracle@oraadserver:/u01/app/oracle/admin/prod20/broker_files/config_db21/callout/ [DB21 (CDB$ROOT)] ls -l
total 12
-rwx------. 1 oracle oinstall 882 Feb  1 16:11 fsfocallout.ora
-rwx------. 1 oracle oinstall  37 Feb  1 16:48 fsfo_postcallout
-rwx------. 1 oracle oinstall  38 Feb  1 16:47 fsfo_precallout
oracle@oraadserver:/u01/app/oracle/admin/prod20/broker_files/config_db21/callout/ [DB21 (CDB$ROOT)]

Let’s see the contents of each file

fsfocallout.ora : fsfo callout configuration file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[oracle@oraadserver3 callout]$ cat fsfocallout.ora
# This is a fast-start failover configuration file.
# The pre-callout script that is run before fast-start failover is enabled.
FastStartFailoverPreCallout=fsfo_precallout
# The timeout value (in seconds) for pre-callout script
#FastStartFailoverPreCalloutTimeout=1200
FastStartFailoverPreCalloutTimeout=25
# The name of the suc file created by the pre-callout script.
FastStartFailoverPreCalloutSucFileName=fsfo_precallout.suc
# The name of the error file that the pre-callout script creates
FastStartFailoverPreCalloutErrorFileName=fsfo_precallout.err
# Action taken by observer if the suc file does not exist after FastStartFailoverPreCalloutTimeout seconds
# or if an error file is detected before FastStartFailoverPreCalloutTimeout seconds passed
FastStartFailoverActionOnPreCalloutFailure=STOP
#FastStartFailoverActionOnPreCalloutFailure=CONTINUE
# The post-callout script that is run after fast-start failover succeeds
FastStartFailoverPostCallout=fsfo_postcallout
[oracle@oraadserver3 callout]$

fsfo_precallout : to be executed before the FSFO

1
2
3
4
5
6
7
8
9
10
11
[oracle@oraadserver3 callout]$ cat fsfo_precallout
#! /bin/bash
if [ 1 -lt 100 ]
 then
   touch /tmp/test
   echo "starting fun observer" > /tmp/test
   touch /u01/app/oracle/admin/prod20/broker_files/config_db21/callout/fsfo_precallout.suc
else
  touch /u01/app/oracle/admin/prod20/broker_files/config_db21/callout/fsfo_precallout.err
fi
[oracle@oraadserver3 callout]$

fsfo_postcallout : to be executed after the FSFO

1
2
3
4
5
[oracle@oraadserver3 callout]$ cat fsfo_postcallout
#! /bin/bash
touch /tmp/test_2
echo "end fun observer" >> /tmp/test_2
[oracle@oraadserver3 callout]$

Now let’s simulate a FSFO and let’s see if the pre-callout and post-callout tasks are executed

For this we will execute a shut abort of the primary database

1
2
3
4
5
6
7
8
9
SQL> select db_unique_name from v$database;
DB_UNIQUE_NAME
------------------------------
db21_site1
SQL> shut abort
ORACLE instance shut down.
SQL>

In the observer logfile, we can see following

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
[oracle@oraadserver3 callout]$ cat /u01/app/oracle/admin/prod20/broker_files/config_db21/log/observer_oraadserver3.log
1
[W000 2022-03-04T17:34:19.533+01:00] Primary database cannot be reached.
[W000 2022-03-04T17:34:19.534+01:00] Fast-Start Failover threshold has not exceeded. Retry for the next 15 seconds
[W000 2022-03-04T17:34:20.534+01:00] Try to connect to the primary.
[W000 2022-03-04T17:34:33.053+01:00] Primary database cannot be reached.
[W000 2022-03-04T17:34:33.053+01:00] Fast-Start Failover threshold has not exceeded. Retry for the next 1 second
[W000 2022-03-04T17:34:34.064+01:00] Try to connect to the primary.
[W000 2022-03-04T17:34:36.084+01:00] Primary database cannot be reached.
[W000 2022-03-04T17:34:36.084+01:00] Fast-Start Failover threshold has expired.
[W000 2022-03-04T17:34:36.085+01:00] Succeeded to parse FSFO callout config file '/u01/app/oracle/admin/prod20/broker_files/config_db21/callout/fsfocallout.ora'
[W000 2022-03-04T17:34:36.085+01:00] Try to connect to the standby.
[W000 2022-03-04T17:34:36.085+01:00] Check if the standby is ready for failover.
[W000 2022-03-04T17:34:36.106+01:00] Doing pre-FSFO callout.
[W000 2022-03-04T17:34:37.107+01:00] Found the pre-FSFO callout suc file '/u01/app/oracle/admin/prod20/broker_files/config_db21/callout/fsfo_precallout.suc'
[W000 2022-03-04T17:34:37.107+01:00] Will continue Fast-Start Failover.
[S025 2022-03-04T17:34:37.107+01:00] Fast-Start Failover started...
2022-03-04T17:34:37.107+01:00
Initiating Fast-Start Failover to database "DB21_SITE2"...
[S025 2022-03-04T17:34:37.108+01:00] Initiating Fast-start Failover.
2022-03-04T17:34:37.124+01:00
Performing failover NOW, please wait...
2022-03-04T17:34:57.927+01:00
Failover succeeded, new primary is "DB21_SITE2".
2022-03-04T17:34:57.927+01:00
Failover processing complete, broker ready.
2022-03-04T17:34:57.927+01:00
[S025 2022-03-04T17:34:57.927+01:00] Fast-Start Failover finished...
[W000 2022-03-04T17:34:57.927+01:00] Failover succeeded. Restart pinging.
[W000 2022-03-04T17:34:57.937+01:00] Primary database has changed to DB21_SITE2.
[W000 2022-03-04T17:34:57.940+01:00] Try to connect to the primary.
…
…
[W000 2022-03-04T17:35:08.317+01:00] Failed to ping the new standby.
[W000 2022-03-04T17:35:08.317+01:00] Try to connect to the new standby DB21_SITE1.
[W000 2022-03-04T17:35:11.317+01:00] Connection to the new standby restored!
[W000 2022-03-04T17:35:11.331+01:00] Failed to ping the new standby.
[W000 2022-03-04T17:35:12.345+01:00] Try to connect to the new standby DB21_SITE1.
[W000 2022-03-04T17:35:15.331+01:00] Connection to the new standby restored!
[W000 2022-03-04T17:35:17.322+01:00] Failed to ping the new standby.
[W000 2022-03-04T17:35:17.322+01:00] Try to connect to the new standby DB21_SITE1.
[W000 2022-03-04T17:35:23.322+01:00] Connection to the new standby restored!
[W000 2022-03-04T17:35:39.344+01:00] Try to connect to the primary DB21_SITE2.
[W000 2022-03-04T17:35:41.324+01:00] Connection to the primary restored!
[W000 2022-03-04T17:35:41.324+01:00] Wait for new primary to be ready to reinstate.
[W000 2022-03-04T17:35:42.342+01:00] New primary is now ready to reinstate.
[W000 2022-03-04T17:35:43.342+01:00] Issuing REINSTATE command.
[oracle@oraadserver3 callout]$

In the callout directory we have the .suc file

1
2
3
[oracle@oraadserver3 callout]$ ls -ltra *.suc
-rw-r--r--. 1 oracle oinstall 0 Mar  4 17:34 fsfo_precallout.suc
[oracle@oraadserver3 callout]$

And in /tmp directory

1
2
3
4
[oracle@oraadserver3 callout]$ ls -l /tmp/test*
-rw-r--r--. 1 oracle oinstall 22 Mar  4 17:34 /tmp/test
-rw-r--r--. 1 oracle oinstall 17 Mar  4 17:34 /tmp/test_2
[oracle@oraadserver3 callout]$
1
2
3
4
5
[oracle@oraadserver3 callout]$ cat /tmp/test
starting fun observer
[oracle@oraadserver3 callout]$ cat /tmp/test_2
end fun observer
[oracle@oraadserver3 callout]$

Conclusion

Sure, that this feature will help (configuring clients before FSFO, stopping application ….)

Post Views: 2,615
Thumbnail [60x60]
by
Oracle Team