Skip to content

Oracle 21c Broker : Fast-Start Failover Callout Scripts in Action

By Mouhamadou Diaw

With Oracle 21c (In fact Oracle 20c), we have a nice new feature with the broker which is the Fast-start Failover Callout Configuration scripts.
I talked about this feature in some presentations I did about the new features with Oracle 21c for the broker
This feature will allow to automate tasks that must be performed before and after a fast-start failover operation.

Yes this means that with Oracle 21c, we can configure some tasks to be executed before and after doing a fast_start failover. Sure that this feature will make life easier.

The use of this feature is very simple

We have to configure the DG_ADMIN variable which will be the default location of Client-side Broker Files. See my previous blog

The callout scripts should be located in the $DG_ADMIN/config_ConfigurationSimpleName/callout.
The configuration file must be named fsfocallout.ora, and it’s mandatory.
This fsfocallout.ora file will contain the names of the pre-callout configuration script and post-callout configuration script.

The pre_callout script must create a .suc file and a .err file. Before doing a FSFO, then the observer will check for .suc file or a for a .err file. If a .suc file exists then the observer will initiate a FSFO.

The .suc file and the .err file must be at the same location that the callout scripts.

Below the configuration I am using

oraadserver : 192.168.2.21 for the primary database
oraadserver2 : 192.168.2.22 for the standby database
oraadserver3 : 192.168.2.23 for the observer

The FSFO is already configured

1
2
3
4
5
6
7
8
9
10
11
12
13
DGMGRL> show configuration
Configuration - db21
  Protection Mode: MaxPerformance
  Members:
  DB21_SITE1 - Primary database
    DB21_SITE2 - (*) Physical standby database
Fast-Start Failover: Enabled in Potential Data Loss Mode
Configuration Status:
SUCCESS   (status updated 5 seconds ago)

And the observer is running

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
DGMGRL> show observer
Configuration - db21
  Fast-Start Failover:     ENABLED
  Primary:            DB21_SITE1
  Active Target:      DB21_SITE2
Observer "oraadserver3" - Master
  Host Name:                    oraadserver3
  Last Ping to Primary:         2 seconds ago
  Last Ping to Target:          0 seconds ago
  Log File:                     /u01/app/oracle/admin/prod20/broker_files/config_db21/log/observer_oraadserver3.log
  State File:                   /u01/app/oracle/admin/prod20/broker_files/config_db21/dat/fsfo.dat
DGMGRL>

On my observer server, the $DG_ADMIN variable is configured (directories of the DG_ADMIN must have some prerequisites (see previous blog or the documentation )

1
2
3
[oracle@oraadserver3 callout]$ echo $DG_ADMIN/
/u01/app/oracle/admin/prod20/broker_files/
[oracle@oraadserver3 callout]$

And in the $DG_ADMIN

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
[oracle@oraadserver3 callout]$ ls -lR $DG_ADMIN/
/u01/app/oracle/admin/prod20/broker_files/:
total 0
drwx------. 2 oracle oinstall  6 Feb  3  2021 admin
drwx------. 5 oracle oinstall 40 Feb 25 11:35 config_db21
/u01/app/oracle/admin/prod20/broker_files/admin:
total 0
/u01/app/oracle/admin/prod20/broker_files/config_db21:
total 4
drwx------. 2 oracle oinstall   99 Mar  4 16:36 callout
drwx------. 2 oracle oinstall   21 Nov 29 13:50 dat
drwx------. 2 oracle oinstall 4096 Feb 25 13:35 log
/u01/app/oracle/admin/prod20/broker_files/config_db21/callout:
total 12
-rwx------. 1 oracle oinstall 979 Mar  4 16:24 fsfocallout.ora
-rwx------. 1 oracle oinstall  57 Feb 25 11:36 fsfo_postcallout
-rwx------. 1 oracle oinstall 351 Mar  4 16:35 fsfo_precallout
-rw-r--r--. 1 oracle oinstall   0 Mar  4 16:36 fsfo_precallout.suc
/u01/app/oracle/admin/prod20/broker_files/config_db21/dat:
total 12
-rw-r--r--. 1 oracle oinstall 8344 Mar  4 16:48 fsfo.dat
/u01/app/oracle/admin/prod20/broker_files/config_db21/log:
total 84
-rw-r-----. 1 oracle oinstall 66093 Mar  4 16:48 observer_oraadserver3.log
[oracle@oraadserver3 callout]$

In the callout directory, we have the FSFO callout scripts

1
2
3
4
5
6
7
8
oracle@oraadserver:/u01/app/oracle/admin/prod20/broker_files/config_db21/callout/ [DB21 (CDB$ROOT)] pwd
/u01/app/oracle/admin/prod20/broker_files/config_db21/callout
oracle@oraadserver:/u01/app/oracle/admin/prod20/broker_files/config_db21/callout/ [DB21 (CDB$ROOT)] ls -l
total 12
-rwx------. 1 oracle oinstall 882 Feb  1 16:11 fsfocallout.ora
-rwx------. 1 oracle oinstall  37 Feb  1 16:48 fsfo_postcallout
-rwx------. 1 oracle oinstall  38 Feb  1 16:47 fsfo_precallout
oracle@oraadserver:/u01/app/oracle/admin/prod20/broker_files/config_db21/callout/ [DB21 (CDB$ROOT)]

Let’s see the contents of each file

fsfocallout.ora : fsfo callout configuration file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[oracle@oraadserver3 callout]$ cat fsfocallout.ora
# This is a fast-start failover configuration file.
# The pre-callout script that is run before fast-start failover is enabled.
FastStartFailoverPreCallout=fsfo_precallout
# The timeout value (in seconds) for pre-callout script
#FastStartFailoverPreCalloutTimeout=1200
FastStartFailoverPreCalloutTimeout=25
# The name of the suc file created by the pre-callout script.
FastStartFailoverPreCalloutSucFileName=fsfo_precallout.suc
# The name of the error file that the pre-callout script creates
FastStartFailoverPreCalloutErrorFileName=fsfo_precallout.err
# Action taken by observer if the suc file does not exist after FastStartFailoverPreCalloutTimeout seconds
# or if an error file is detected before FastStartFailoverPreCalloutTimeout seconds passed
FastStartFailoverActionOnPreCalloutFailure=STOP
#FastStartFailoverActionOnPreCalloutFailure=CONTINUE
# The post-callout script that is run after fast-start failover succeeds
FastStartFailoverPostCallout=fsfo_postcallout
[oracle@oraadserver3 callout]$

fsfo_precallout : to be executed before the FSFO

1
2
3
4
5
6
7
8
9
10
11
[oracle@oraadserver3 callout]$ cat fsfo_precallout
#! /bin/bash
if [ 1 -lt 100 ]
 then
   touch /tmp/test
   echo "starting fun observer" > /tmp/test
   touch /u01/app/oracle/admin/prod20/broker_files/config_db21/callout/fsfo_precallout.suc
else
  touch /u01/app/oracle/admin/prod20/broker_files/config_db21/callout/fsfo_precallout.err
fi
[oracle@oraadserver3 callout]$

fsfo_postcallout : to be executed after the FSFO

1
2
3
4
5
[oracle@oraadserver3 callout]$ cat fsfo_postcallout
#! /bin/bash
touch /tmp/test_2
echo "end fun observer" >> /tmp/test_2
[oracle@oraadserver3 callout]$

Now let’s simulate a FSFO and let’s see if the pre-callout and post-callout tasks are executed

For this we will execute a shut abort of the primary database

1
2
3
4
5
6
7
8
9
SQL> select db_unique_name from v$database;
DB_UNIQUE_NAME
------------------------------
db21_site1
SQL> shut abort
ORACLE instance shut down.
SQL>

In the observer logfile, we can see following

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
[oracle@oraadserver3 callout]$ cat /u01/app/oracle/admin/prod20/broker_files/config_db21/log/observer_oraadserver3.log
1
[W000 2022-03-04T17:34:19.533+01:00] Primary database cannot be reached.
[W000 2022-03-04T17:34:19.534+01:00] Fast-Start Failover threshold has not exceeded. Retry for the next 15 seconds
[W000 2022-03-04T17:34:20.534+01:00] Try to connect to the primary.
[W000 2022-03-04T17:34:33.053+01:00] Primary database cannot be reached.
[W000 2022-03-04T17:34:33.053+01:00] Fast-Start Failover threshold has not exceeded. Retry for the next 1 second
[W000 2022-03-04T17:34:34.064+01:00] Try to connect to the primary.
[W000 2022-03-04T17:34:36.084+01:00] Primary database cannot be reached.
[W000 2022-03-04T17:34:36.084+01:00] Fast-Start Failover threshold has expired.
[W000 2022-03-04T17:34:36.085+01:00] Succeeded to parse FSFO callout config file '/u01/app/oracle/admin/prod20/broker_files/config_db21/callout/fsfocallout.ora'
[W000 2022-03-04T17:34:36.085+01:00] Try to connect to the standby.
[W000 2022-03-04T17:34:36.085+01:00] Check if the standby is ready for failover.
[W000 2022-03-04T17:34:36.106+01:00] Doing pre-FSFO callout.
[W000 2022-03-04T17:34:37.107+01:00] Found the pre-FSFO callout suc file '/u01/app/oracle/admin/prod20/broker_files/config_db21/callout/fsfo_precallout.suc'
[W000 2022-03-04T17:34:37.107+01:00] Will continue Fast-Start Failover.
[S025 2022-03-04T17:34:37.107+01:00] Fast-Start Failover started...
2022-03-04T17:34:37.107+01:00
Initiating Fast-Start Failover to database "DB21_SITE2"...
[S025 2022-03-04T17:34:37.108+01:00] Initiating Fast-start Failover.
2022-03-04T17:34:37.124+01:00
Performing failover NOW, please wait...
2022-03-04T17:34:57.927+01:00
Failover succeeded, new primary is "DB21_SITE2".
2022-03-04T17:34:57.927+01:00
Failover processing complete, broker ready.
2022-03-04T17:34:57.927+01:00
[S025 2022-03-04T17:34:57.927+01:00] Fast-Start Failover finished...
[W000 2022-03-04T17:34:57.927+01:00] Failover succeeded. Restart pinging.
[W000 2022-03-04T17:34:57.937+01:00] Primary database has changed to DB21_SITE2.
[W000 2022-03-04T17:34:57.940+01:00] Try to connect to the primary.
…
…
[W000 2022-03-04T17:35:08.317+01:00] Failed to ping the new standby.
[W000 2022-03-04T17:35:08.317+01:00] Try to connect to the new standby DB21_SITE1.
[W000 2022-03-04T17:35:11.317+01:00] Connection to the new standby restored!
[W000 2022-03-04T17:35:11.331+01:00] Failed to ping the new standby.
[W000 2022-03-04T17:35:12.345+01:00] Try to connect to the new standby DB21_SITE1.
[W000 2022-03-04T17:35:15.331+01:00] Connection to the new standby restored!
[W000 2022-03-04T17:35:17.322+01:00] Failed to ping the new standby.
[W000 2022-03-04T17:35:17.322+01:00] Try to connect to the new standby DB21_SITE1.
[W000 2022-03-04T17:35:23.322+01:00] Connection to the new standby restored!
[W000 2022-03-04T17:35:39.344+01:00] Try to connect to the primary DB21_SITE2.
[W000 2022-03-04T17:35:41.324+01:00] Connection to the primary restored!
[W000 2022-03-04T17:35:41.324+01:00] Wait for new primary to be ready to reinstate.
[W000 2022-03-04T17:35:42.342+01:00] New primary is now ready to reinstate.
[W000 2022-03-04T17:35:43.342+01:00] Issuing REINSTATE command.
[oracle@oraadserver3 callout]$

In the callout directory we have the .suc file

1
2
3
[oracle@oraadserver3 callout]$ ls -ltra *.suc
-rw-r--r--. 1 oracle oinstall 0 Mar  4 17:34 fsfo_precallout.suc
[oracle@oraadserver3 callout]$

And in /tmp directory

1
2
3
4
[oracle@oraadserver3 callout]$ ls -l /tmp/test*
-rw-r--r--. 1 oracle oinstall 22 Mar  4 17:34 /tmp/test
-rw-r--r--. 1 oracle oinstall 17 Mar  4 17:34 /tmp/test_2
[oracle@oraadserver3 callout]$
1
2
3
4
5
[oracle@oraadserver3 callout]$ cat /tmp/test
starting fun observer
[oracle@oraadserver3 callout]$ cat /tmp/test_2
end fun observer
[oracle@oraadserver3 callout]$

Conclusion

Sure, that this feature will help (configuring clients before FSFO, stopping application ….)

Let’s work together

Ready to operate at
the next level?

Tell us about your challenge. We’ll bring the depth, the expertise, and the Swiss precision.

Oracle Team

Oracle Team

Posts published under the Oracle Team account are collective contributions by the dbi services Oracle consultants. The practice covers the full Oracle Database lifecycle: architecture, migration and upgrade, high availability with RAC and Data Guard, backup and recovery, performance tuning, and engineered systems such as Oracle Database Appliance and Exadata. The team holds Oracle certifications up to Certified Master level and operates customer platforms under service level agreements.View Author posts

1,009 posts since December 2010

Most read in Oracle

The Oracle posts our readers open most: the all-time favourites, and what is trending right now.

Leave a Comment

Your email address will not be published. Required fields are marked *