With Oracle 21c (In fact Oracle 20c), we have a nice new feature with the broker which is the Fast-start Failover Callout Configuration scripts.
I talked about this feature in some presentations I did about the new features with Oracle 21c for the broker
This feature will allow to automate tasks that must be performed before and after a fast-start failover operation.

Yes this means that with Oracle 21c, we can configure some tasks to be executed before and after doing a fast_start failover. Sure that this feature will make life easier.

The use of this feature is very simple

We have to configure the DG_ADMIN variable which will be the default location of Client-side Broker Files. See my previous blog

The callout scripts should be located in the $DG_ADMIN/config_ConfigurationSimpleName/callout.
The configuration file must be named fsfocallout.ora, and it’s mandatory.
This fsfocallout.ora file will contain the names of the pre-callout configuration script and post-callout configuration script.

The pre_callout script must create a .suc file and a .err file. Before doing a FSFO, then the observer will check for .suc file or a for a .err file. If a .suc file exists then the observer will initiate a FSFO.

The .suc file and the .err file must be at the same location that the callout scripts.

Below the configuration I am using

oraadserver : for the primary database
oraadserver2 : for the standby database
oraadserver3 : for the observer

The FSFO is already configured

DGMGRL> show configuration

Configuration - db21

  Protection Mode: MaxPerformance
  DB21_SITE1 - Primary database
    DB21_SITE2 - (*) Physical standby database

Fast-Start Failover: Enabled in Potential Data Loss Mode

Configuration Status:
SUCCESS   (status updated 5 seconds ago)

And the observer is running

DGMGRL> show observer

Configuration - db21

  Fast-Start Failover:     ENABLED

  Primary:            DB21_SITE1
  Active Target:      DB21_SITE2

Observer "oraadserver3" - Master

  Host Name:                    oraadserver3
  Last Ping to Primary:         2 seconds ago
  Last Ping to Target:          0 seconds ago
  Log File:                     /u01/app/oracle/admin/prod20/broker_files/config_db21/log/observer_oraadserver3.log
  State File:                   /u01/app/oracle/admin/prod20/broker_files/config_db21/dat/fsfo.dat


On my observer server, the $DG_ADMIN variable is configured (directories of the DG_ADMIN must have some prerequisites (see previous blog or the documentation )

[[email protected] callout]$ echo $DG_ADMIN/
[[email protected] callout]$

And in the $DG_ADMIN

[[email protected] callout]$ ls -lR $DG_ADMIN/
total 0
drwx------. 2 oracle oinstall  6 Feb  3  2021 admin
drwx------. 5 oracle oinstall 40 Feb 25 11:35 config_db21

total 0

total 4
drwx------. 2 oracle oinstall   99 Mar  4 16:36 callout
drwx------. 2 oracle oinstall   21 Nov 29 13:50 dat
drwx------. 2 oracle oinstall 4096 Feb 25 13:35 log

total 12
-rwx------. 1 oracle oinstall 979 Mar  4 16:24 fsfocallout.ora
-rwx------. 1 oracle oinstall  57 Feb 25 11:36 fsfo_postcallout
-rwx------. 1 oracle oinstall 351 Mar  4 16:35 fsfo_precallout
-rw-r--r--. 1 oracle oinstall   0 Mar  4 16:36 fsfo_precallout.suc

total 12
-rw-r--r--. 1 oracle oinstall 8344 Mar  4 16:48 fsfo.dat

total 84
-rw-r-----. 1 oracle oinstall 66093 Mar  4 16:48 observer_oraadserver3.log
[[email protected] callout]$

In the callout directory, we have the FSFO callout scripts

[email protected]:/u01/app/oracle/admin/prod20/broker_files/config_db21/callout/ [DB21 (CDB$ROOT)] pwd
[email protected]:/u01/app/oracle/admin/prod20/broker_files/config_db21/callout/ [DB21 (CDB$ROOT)] ls -l
total 12
-rwx------. 1 oracle oinstall 882 Feb  1 16:11 fsfocallout.ora
-rwx------. 1 oracle oinstall  37 Feb  1 16:48 fsfo_postcallout
-rwx------. 1 oracle oinstall  38 Feb  1 16:47 fsfo_precallout
[email protected]:/u01/app/oracle/admin/prod20/broker_files/config_db21/callout/ [DB21 (CDB$ROOT)]

Let’s see the contents of each file

fsfocallout.ora : fsfo callout configuration file

[[email protected] callout]$ cat fsfocallout.ora
# This is a fast-start failover configuration file.

# The pre-callout script that is run before fast-start failover is enabled.

# The timeout value (in seconds) for pre-callout script

# The name of the suc file created by the pre-callout script.

# The name of the error file that the pre-callout script creates

# Action taken by observer if the suc file does not exist after FastStartFailoverPreCalloutTimeout seconds
# or if an error file is detected before FastStartFailoverPreCalloutTimeout seconds passed

# The post-callout script that is run after fast-start failover succeeds
[[email protected] callout]$

fsfo_precallout : to be executed before the FSFO

[[email protected] callout]$ cat fsfo_precallout
#! /bin/bash
if [ 1 -lt 100 ]
   touch /tmp/test
   echo "starting fun observer" > /tmp/test
   touch /u01/app/oracle/admin/prod20/broker_files/config_db21/callout/fsfo_precallout.suc
  touch /u01/app/oracle/admin/prod20/broker_files/config_db21/callout/fsfo_precallout.err
[[email protected] callout]$

fsfo_postcallout : to be executed after the FSFO

[[email protected] callout]$ cat fsfo_postcallout
#! /bin/bash
touch /tmp/test_2
echo "end fun observer" >> /tmp/test_2
[[email protected] callout]$

Now let’s simulate a FSFO and let’s see if the pre-callout and post-callout tasks are executed

For this we will execute a shut abort of the primary database

SQL> select db_unique_name from v$database;


SQL> shut abort
ORACLE instance shut down.

In the observer logfile, we can see following

[[email protected] callout]$ cat /u01/app/oracle/admin/prod20/broker_files/config_db21/log/observer_oraadserver3.log
[W000 2022-03-04T17:34:19.533+01:00] Primary database cannot be reached.
[W000 2022-03-04T17:34:19.534+01:00] Fast-Start Failover threshold has not exceeded. Retry for the next 15 seconds
[W000 2022-03-04T17:34:20.534+01:00] Try to connect to the primary.
[W000 2022-03-04T17:34:33.053+01:00] Primary database cannot be reached.
[W000 2022-03-04T17:34:33.053+01:00] Fast-Start Failover threshold has not exceeded. Retry for the next 1 second
[W000 2022-03-04T17:34:34.064+01:00] Try to connect to the primary.
[W000 2022-03-04T17:34:36.084+01:00] Primary database cannot be reached.
[W000 2022-03-04T17:34:36.084+01:00] Fast-Start Failover threshold has expired.
[W000 2022-03-04T17:34:36.085+01:00] Succeeded to parse FSFO callout config file '/u01/app/oracle/admin/prod20/broker_files/config_db21/callout/fsfocallout.ora'
[W000 2022-03-04T17:34:36.085+01:00] Try to connect to the standby.
[W000 2022-03-04T17:34:36.085+01:00] Check if the standby is ready for failover.
[W000 2022-03-04T17:34:36.106+01:00] Doing pre-FSFO callout.
[W000 2022-03-04T17:34:37.107+01:00] Found the pre-FSFO callout suc file '/u01/app/oracle/admin/prod20/broker_files/config_db21/callout/fsfo_precallout.suc'
[W000 2022-03-04T17:34:37.107+01:00] Will continue Fast-Start Failover.
[S025 2022-03-04T17:34:37.107+01:00] Fast-Start Failover started...

Initiating Fast-Start Failover to database "DB21_SITE2"...
[S025 2022-03-04T17:34:37.108+01:00] Initiating Fast-start Failover.
Performing failover NOW, please wait...

Failover succeeded, new primary is "DB21_SITE2".

Failover processing complete, broker ready.
[S025 2022-03-04T17:34:57.927+01:00] Fast-Start Failover finished...
[W000 2022-03-04T17:34:57.927+01:00] Failover succeeded. Restart pinging.
[W000 2022-03-04T17:34:57.937+01:00] Primary database has changed to DB21_SITE2.
[W000 2022-03-04T17:34:57.940+01:00] Try to connect to the primary.
[W000 2022-03-04T17:35:08.317+01:00] Failed to ping the new standby.
[W000 2022-03-04T17:35:08.317+01:00] Try to connect to the new standby DB21_SITE1.
[W000 2022-03-04T17:35:11.317+01:00] Connection to the new standby restored!
[W000 2022-03-04T17:35:11.331+01:00] Failed to ping the new standby.
[W000 2022-03-04T17:35:12.345+01:00] Try to connect to the new standby DB21_SITE1.
[W000 2022-03-04T17:35:15.331+01:00] Connection to the new standby restored!
[W000 2022-03-04T17:35:17.322+01:00] Failed to ping the new standby.
[W000 2022-03-04T17:35:17.322+01:00] Try to connect to the new standby DB21_SITE1.
[W000 2022-03-04T17:35:23.322+01:00] Connection to the new standby restored!
[W000 2022-03-04T17:35:39.344+01:00] Try to connect to the primary DB21_SITE2.
[W000 2022-03-04T17:35:41.324+01:00] Connection to the primary restored!
[W000 2022-03-04T17:35:41.324+01:00] Wait for new primary to be ready to reinstate.
[W000 2022-03-04T17:35:42.342+01:00] New primary is now ready to reinstate.
[W000 2022-03-04T17:35:43.342+01:00] Issuing REINSTATE command.

[[email protected] callout]$

In the callout directory we have the .suc file

[[email protected] callout]$ ls -ltra *.suc
-rw-r--r--. 1 oracle oinstall 0 Mar  4 17:34 fsfo_precallout.suc
[[email protected] callout]$

And in /tmp directory

[[email protected] callout]$ ls -l /tmp/test*
-rw-r--r--. 1 oracle oinstall 22 Mar  4 17:34 /tmp/test
-rw-r--r--. 1 oracle oinstall 17 Mar  4 17:34 /tmp/test_2
[[email protected] callout]$
[[email protected] callout]$ cat /tmp/test
starting fun observer
[[email protected] callout]$ cat /tmp/test_2
end fun observer
[[email protected] callout]$


Sure, that this feature will help (configuring clients before FSFO, stopping application ….)