Oracle Database Appliance (ODA): Some listeners do not start anymore when booting

A colleague of mine had the issue some time ago and I recently experienced the same behaviour at a customer’s ODA: A couple of Listeners did not start when booting. Starting them as user grid with

srvctl start listener -listener <listener-name>

worked without any problem though. So what is the issue?

Analysis

Looking at the listeners which do not start in the listener.ora it becomes obvious that all listeners, which had an SID_LIST_<listener-name> did not start when booting.

My first step was to check crs:

[root@mysrv ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
ORACLE_HOME = [/home/oracle] ? /u01/app/19.20.0.0/grid
The Oracle base has been set to /u01/app/grid
[root@mysrv ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

That looks good.

For testing purposes it’s of course easier to stop crs and restart it instead of rebooting the server:

[root@mysrv ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'mysrv'
CRS-2673: Attempting to stop 'ora.crsd' on 'mysrv'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'mysrv'
...
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'mysrv' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@mysrv ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@mysrv ~]# 

The listeners could be shutdown without any issue, but they did not come up when starting crs. In the alert.log I did see this:

[root@mysrv app]# view /u01/app/grid/diag/crs/mysrv/crs/trace/alert.log
....
2023-10-24 17:04:57.042 [CRSD(90854)]CRS-2807: Resource 'ora.LISTENER1.lsnr' failed to start automatically.
2023-10-24 17:04:57.044 [CRSD(90854)]CRS-2807: Resource 'ora.LISTENER2.lsnr' failed to start automatically.
...

To find the reason why the cluster registry services of grid infrastructure did not start those specific listeners LISTENER1 and LISTENER2, I searched in the oraagent trace-files for SID_LIST_LISTENER1:

[root@mysrv app]# pwd
/u01/app
[root@mysrv app]# grep -il sid_list_listener1 ./grid/diag/crs/mysrv/crs/trace/crsd_oraagent_*.trc
...
./grid/diag/crs/mysrv/crs/trace/crsd_oraagent_grid.trc
...
[root@mysrv app]# view ./grid/diag/crs/mysrv/crs/trace/crsd_oraagent_grid.trc
...
2023-10-24 20:46:58.914 :CLSDYNAM:1738524416: [ora.LISTENER1.lsnr]{1:60057:2} [start] ConfigFile::parse mmap name:sid_list_listener1 nameWithCase:SID_LIST_LISTENER1 value: comment:
2023-10-24 20:46:58.915 :CLSDYNAM:1700701952: [ora.LISTENER1.lsnr]{1:60057:2} [start] LsnrAgent::Lsnrctl::start lsnrctl start complete lsnrname:LISTENER1 output:
LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 24-OCT-2023 20:46:58

Copyright (c) 1991, 2023, Oracle.  All rights reserved.

Starting /u01/app/19.20.0.0/grid/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 19.0.0.0.0 - Production
System parameter file is /u01/app/19.20.0.0/grid/network/admin/grid/listener.ora
Log messages written to /u01/app/grid/diag/tnslsnr/mysrv/listener1/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER1)))
TNS-01201: Listener cannot find executable /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_10/bin/oracle for SID INST9

Listener failed to start. See the error message(s) above...

So the listener could not find the oracle executable in the ORACLE_HOME provided in the SID_LIST_LISTENER1. The executable is of course required to start a connection through a statistic listener entry. When I saw that, it became clear why the listeners did not start: Since ODA 19.11. the ORACLE_HOMEs are located on the ACFS-Filesystem

/u01/app/odaorahome

REMARK: For more details about that change see an earlier Blog.

The issue was that that filesystem has not been mounted yet when the listener is being started by the oraagent. So I needed to add a startup dependency to the listeners to wait until the ACFS-Filesystem has been mounted.

REMARK: Starting the listener with srvctl manually worked, because in the meantime the ACFS-filesystem was mounted.

Solution and verification of the solution

What is the current start dependency?

[grid@mysrv grid]$ crsctl status resource ora.LISTENER1.lsnr -f | grep START_DEPENDENCIES=
START_DEPENDENCIES=hard(type:ora.cluster_vip_net1.type) pullup(type:ora.cluster_vip_net1.type)
[grid@mysrv grid]$

How to modify the start dependency? Oracle provides the command crsctl modify, but it allows modifications only with the “-unsupported” option set. We opened a SR with Oracle and they confirmed that we can use that option. The resource we are depending on is “ora.data.orahome_sh.acfs”. So the needed adjustment is:

[grid@mysrv grid]$ crsctl modify resource ora.LISTENER1.lsnr -attr "START_DEPENDENCIES='hard(type:ora.cluster_vip_net1.type,ora.data.orahome_sh.acfs) pullup(type:ora.cluster_vip_net1.type)'" -unsupported
[grid@mysrv grid]$ 

I tested it (crsctl stop crs -f and crsctl start crs) and it worked, the listener came up as expected. Afterwards I changed all required listeners accordingly and rebooted the server. All listeners came up during the boot process.

Summary

Introducing ORACLE_HOMEs on ACFS in 19.11. was a very welcome improvement, because it took away the space pressure we usually had on ODAs on the /u01-filesystem with many ORACLE_HOMEs. Such new features however, may have side effects, which are not obvious initially. Doing a proper analysis usually yields to a solution or a workaround. If you have to implement the solution/workaround then it’s a good practice to double check with Oracle Support that your solution is supported by Oracle – especially on the ODA.