I was recently patching several Oracle Database Appliance (ODA) X11-2-HA at customer from 19.26 to 19.30 and then finally to 19.32.

During one patching hop I faced an issue where /root became full. One of the consequences was that I had to manually patch local controller firmware. Through this blog I would like to share with you my experience about this problem and the solution I applied.

Problem description

I was at the step to run the update-servercomponents patching:

[root@node0 ~]# /opt/oracle/dcs/bin/odacli update-servercomponents -v 19.30.0.0.0
{
  "jobId" : "03c81877-9d76-426d-b6c8-a7da09ffca66",
  "status" : "Created",
  "message" : "A server update job has been initiated. If a reboot is required, it will be performed as part of the update process. The final status will be provided once the update operation has been completed on all nodes.",
  "reports" : [ ],
  "createTimestamp" : "September 24, 2026 10:11:14 CEST",
  "resourceList" : [ ],
  "description" : "Server Patching to 19.30.0.0.0",
  "updatedTime" : "September 24, 2026 10:11:14 CEST",
  "jobType" : null,
  "cpsMetadata" : null,
  "ecJobId" : null
}

The job patching activity was on-going correctly:

[root@node0 ~]# odacli describe-job -i 03c81877-9d76-426d-b6c8-a7da09ffca66

Job details
----------------------------------------------------------------
                     ID:  03c81877-9d76-426d-b6c8-a7da09ffca66
            Description:  Server Patching to 19.30.0.0.0
                 Status:  Running
                Created:  September 24, 2026 10:11:14 CEST
                Message:

Task Name                                Node Name                 Start Time                               End Time                                 Status
---------------------------------------- ------------------------- ---------------------------------------- ---------------------------------------- ----------------
Validate ILOM server reachable           node0                 September 24, 2026 10:11:24 CEST         September 24, 2026 10:11:24 CEST         Success
Validate ILOM server reachable           node1                 September 24, 2026 10:11:24 CEST         September 24, 2026 10:11:24 CEST         Success
Validating GI user metadata              node0                 September 24, 2026 10:11:24 CEST         September 24, 2026 10:11:24 CEST         Success
Validating GI user metadata              node1                 September 24, 2026 10:11:24 CEST         September 24, 2026 10:11:24 CEST         Success
Modify BM udev rules                     node0                 September 24, 2026 10:11:25 CEST         September 24, 2026 10:11:36 CEST         Success
Modify BM udev rules                     node1                 September 24, 2026 10:11:25 CEST         September 24, 2026 10:11:36 CEST         Success
Stop oakd                                node1                 September 24, 2026 10:11:40 CEST         September 24, 2026 10:11:43 CEST         Success
Creating local repository                node0                 September 24, 2026 10:11:43 CEST         September 24, 2026 10:11:45 CEST         Success
OSPatchBaseRepo
Creating local repository                node1                 September 24, 2026 10:11:43 CEST         September 24, 2026 10:11:45 CEST         Success
OSPatchBaseRepo
Updating versionlock plugin              node0                 September 24, 2026 10:11:45 CEST         September 24, 2026 10:11:48 CEST         Success
Updating versionlock plugin              node1                 September 24, 2026 10:11:45 CEST         September 24, 2026 10:11:48 CEST         Success
Applying OS Patches                      node0                 September 24, 2026 10:11:48 CEST                                                  Running
Applying OS Patches                      node1                 September 24, 2026 10:11:48 CEST                                                  Running

[root@node0 ~]#

I’m used to let a tail -f running on the DCS alert log agent, just to watch all things are on-going:

tail -f /opt/oracle/dcs/log/dcs-agent.log

And I could see, the following error:

2026-09-24 10:34:50,775 ERROR [DcsJobScheduler-Worker] [] c.o.d.c.e.ExceptionFactory: Failed to write to file: /tmp/dcsfiles/asm/asm_2026-09-24_10-34-50.0766_303.sql
DcsException{errorHttpCode=InternalError, msg=failed to write to file /tmp/dcsfiles/asm/asm_2026-09-24_10-34-50.0766_303.sql: No space left on device, msgId=12022, causedBy=java.io.IOException: No space left on device}
     at com.oracle.dcs.commons.exception.DcsException$Builder.build(DcsException.java:143)
     at com.oracle.dcs.commons.exception.ExceptionFactory.failWriteFile(ExceptionFactory.java:477)
     at com.oracle.dcs.commons.exception.ExceptionFactory.lambda$failWriteFileSupplier$18(ExceptionFactory.java:493)
     at com.oracle.dcs.commons.exception.ExceptionFactory.throwFailWriteFile(ExceptionFactory.java:832)
     at com.oracle.dcs.commons.utils.CommonsUtils.writeToSqlFile(CommonsUtils.java:2006)
     at com.oracle.dcs.agent.utils.db.SqlPlusHelper.runASMStatements(SqlPlusHelper.java:996)
     at com.oracle.dcs.agent.schedule.jobs.RebalanceMonitoringHelper.getRebalanceInfo(RebalanceMonitoringHelper.java:169)
     at com.oracle.dcs.agent.schedule.jobs.RebalanceMonitoringHelper.monitorRebalance(RebalanceMonitoringHelper.java:293)
     at com.oracle.dcs.agent.schedule.jobs.RebalanceMonitorJob.executeJob(RebalanceMonitorJob.java:100)
     at com.oracle.dcs.agent.schedule.jobs.RebalanceMonitorJob.run(RebalanceMonitorJob.java:62)
     at com.oracle.dcs.agent.schedule.DcsJobScheduler$DcsExecutionTask.run(DcsJobScheduler.java:572)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
     at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException: No space left on device
     at java.io.FileOutputStream.writeBytes(Native Method)
     at java.io.FileOutputStream.write(FileOutputStream.java:326)
     at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
     at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:316)
     at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:149)
     at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233)
     at com.oracle.dcs.commons.utils.CommonsUtils.writeToSqlFile(CommonsUtils.java:2004)
     ... 9 more
2026-09-24 10:34:50,776 DEBUG [DcsJobScheduler-Worker] [] c.o.d.a.s.DcsJobScheduler: afterExecute() : Begin

Resolution

I immediately checked Root File System:

[root@node0 tmp]# df -h /tmp
Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/VolGroupSys-LogVolRoot   30G   30G     0 100% /

And root file system was full! No space left of device!

How is it possible, knowing one of my pre-patching steps is to check /, /u01, /opt, and /boot file systems occupancies, to ensure that for /, /u01 and /opt I have at least 20 GB available space, running:

df -h / /u01 /opt /boot /opt/oracle/oak/pkgrepos/orapkgs/clones

Checking what could have taken all this space, I could see that a folder named oraapp was taking 23GB space in local root file system:

[root@node0 /]# du -sh *
0    bin
257M boot
3.7M core.oclumon.bin.89762
20M  dev
35M  etc
4.0M home
0    lib
0    lib64
16K  lost+found
4.0K media
92G  mnt
0    null
25G  opt
4.0K oracle-ahf-2511100.zip
23G  oraapp
0    proc
575M root
4.1G run
0    sbin
4.0K srv
0    sys
341M tmp
32G  u01
20K  u02
12K  u03
20K  u04
56K  u99
2.3G usr
1.5G var
[root@node0 /]#

This folder with the name of the application was certainly not generated by the ODA patching. Contacting the customer we could realise that he has been writing some hourly synchronization script to sync application data between primary and standby, which is more than ok. And the data were copied on an ACFS File System that was created for this purpose. Which is totally ok. The problem is that the synchronization job was still running during the patching, and the patching will close all ASM resources. Bingo, the synchronization job created then of course locally a directory in root file system. With customer agreement I immediately deleted that directory.

Lesson learned, we updated the patching MOP to have the new synchronization job stopped before patching.

At the end the patching could anyhow run successfully.

[root@node0 ~]# odacli describe-job -i 03c81877-9d76-426d-b6c8-a7da09ffca66

Job details
----------------------------------------------------------------
                     ID:  03c81877-9d76-426d-b6c8-a7da09ffca66
            Description:  Server Patching to 19.30.0.0.0
                 Status:  Success
                Created:  September 24, 2026 10:11:14 CEST
                Message:

Task Name                                Node Name                 Start Time                               End Time                                 Status
---------------------------------------- ------------------------- ---------------------------------------- ---------------------------------------- ----------------
Validate ILOM server reachable           node0                 September 24, 2026 10:11:24 CEST         September 24, 2026 10:11:24 CEST         Success
Validate ILOM server reachable           node1                 September 24, 2026 10:11:24 CEST         September 24, 2026 10:11:24 CEST         Success
Validating GI user metadata              node0                 September 24, 2026 10:11:24 CEST         September 24, 2026 10:11:24 CEST         Success
Validating GI user metadata              node1                 September 24, 2026 10:11:24 CEST         September 24, 2026 10:11:24 CEST         Success
Modify BM udev rules                     node0                 September 24, 2026 10:11:25 CEST         September 24, 2026 10:11:36 CEST         Success
Modify BM udev rules                     node1                 September 24, 2026 10:11:25 CEST         September 24, 2026 10:11:36 CEST         Success
Stop oakd                                node1                 September 24, 2026 10:11:40 CEST         September 24, 2026 10:11:43 CEST         Success
Creating local repository                node0                 September 24, 2026 10:11:43 CEST         September 24, 2026 10:11:45 CEST         Success
OSPatchBaseRepo
Creating local repository                node1                 September 24, 2026 10:11:43 CEST         September 24, 2026 10:11:45 CEST         Success
OSPatchBaseRepo
Updating versionlock plugin              node0                 September 24, 2026 10:11:45 CEST         September 24, 2026 10:11:48 CEST         Success
Updating versionlock plugin              node1                 September 24, 2026 10:11:45 CEST         September 24, 2026 10:11:48 CEST         Success
Applying OS Patches                      node0                 September 24, 2026 10:11:48 CEST         September 24, 2026 10:19:41 CEST         Success
Applying OS Patches                      node1                 September 24, 2026 10:11:48 CEST         September 24, 2026 10:19:24 CEST         Success
Creating local repository HMPPatchRepo   node0                 September 24, 2026 10:19:41 CEST         September 24, 2026 10:19:42 CEST         Success
Creating local repository HMPPatchRepo   node1                 September 24, 2026 10:19:41 CEST         September 24, 2026 10:19:42 CEST         Success
Applying HMP Patches                     node0                 September 24, 2026 10:19:42 CEST         September 24, 2026 10:19:45 CEST         Success
Applying HMP Patches                     node1                 September 24, 2026 10:19:42 CEST         September 24, 2026 10:19:44 CEST         Success
Setting SELinux mode                     node0                 September 24, 2026 10:19:45 CEST         September 24, 2026 10:19:45 CEST         Success
Setting SELinux mode                     node1                 September 24, 2026 10:19:45 CEST         September 24, 2026 10:19:45 CEST         Success
Starting the clusterware                 node1                 September 24, 2026 10:21:16 CEST         September 24, 2026 10:22:34 CEST         Success
Deconfigure AFD                          node1                 September 24, 2026 10:24:12 CEST         September 24, 2026 10:25:52 CEST         Success
Oda-hw-mgmt upgrade                      node0                 September 24, 2026 10:25:53 CEST         September 24, 2026 10:27:50 CEST         Success
Patch location validation                node0                 September 24, 2026 10:25:53 CEST         September 24, 2026 10:25:53 CEST         Success
Patch location validation                node1                 September 24, 2026 10:25:53 CEST         September 24, 2026 10:25:53 CEST         Success
Oda-hw-mgmt upgrade                      node1                 September 24, 2026 10:27:50 CEST         September 24, 2026 10:29:47 CEST         Success
Post Deconfigure AFD                     node1                 September 24, 2026 10:31:22 CEST         September 24, 2026 10:32:43 CEST         Success
OSS Patching                             node0                 September 24, 2026 10:32:43 CEST         September 24, 2026 10:32:44 CEST         Success
OSS Patching                             node1                 September 24, 2026 10:32:43 CEST         September 24, 2026 10:32:44 CEST         Success
Installing SQLcl software                node1                 September 24, 2026 10:32:44 CEST         September 24, 2026 10:32:47 CEST         Success
Applying Firmware local Disk Patches     node0                 September 24, 2026 10:32:47 CEST         September 24, 2026 10:32:51 CEST         Success
Applying Firmware local Disk Patches     node1                 September 24, 2026 10:32:51 CEST         September 24, 2026 10:32:55 CEST         Success
Applying Firmware local Controller Patch node0                 September 24, 2026 10:32:58 CEST         September 24, 2026 10:33:01 CEST         Success
Applying Firmware local Controller Patch node1                 September 24, 2026 10:33:01 CEST         September 24, 2026 10:40:45 CEST         Success
Checking Ilom patch Version              node0                 September 24, 2026 10:40:46 CEST         September 24, 2026 10:40:46 CEST         Success
Checking Ilom patch Version              node1                 September 24, 2026 10:40:46 CEST         September 24, 2026 10:40:46 CEST         Success
Patch location validation                node0                 September 24, 2026 10:40:46 CEST         September 24, 2026 10:40:46 CEST         Success
Patch location validation                node1                 September 24, 2026 10:40:46 CEST         September 24, 2026 10:40:46 CEST         Success
Save password in Wallet                  node1                 September 24, 2026 10:40:47 CEST         September 24, 2026 10:40:49 CEST         Success
Disabling IPMI v2                        node0                 September 24, 2026 10:40:49 CEST         September 24, 2026 10:40:50 CEST         Success
Disabling IPMI v2                        node1                 September 24, 2026 10:40:50 CEST         September 24, 2026 10:40:51 CEST         Success
Apply Ilom patch                         node0                 September 24, 2026 10:40:51 CEST         September 24, 2026 10:52:26 CEST         Success
Apply Ilom patch                         node1                 September 24, 2026 10:52:26 CEST         September 24, 2026 11:03:45 CEST         Success
Copying Flash Bios to Temp location      node0                 September 24, 2026 11:03:45 CEST         September 24, 2026 11:03:45 CEST         Success
Copying Flash Bios to Temp location      node1                 September 24, 2026 11:03:45 CEST         September 24, 2026 11:03:45 CEST         Success
Start oakd                               node1                 September 24, 2026 11:04:04 CEST         September 24, 2026 11:04:20 CEST         Success
Add SYSNAME in Env                       node1                 September 24, 2026 11:04:21 CEST         September 24, 2026 11:04:21 CEST         Success
Cleanup JRE Home                         node0                 September 24, 2026 11:04:21 CEST         September 24, 2026 11:04:21 CEST         Success
Cleanup JRE Home                         node1                 September 24, 2026 11:04:21 CEST         September 24, 2026 11:04:21 CEST         Success
Starting the clusterware                 node1                 September 24, 2026 11:05:51 CEST         September 24, 2026 11:07:08 CEST         Success
Update lvm.conf file                     node0                 September 24, 2026 11:07:09 CEST         September 24, 2026 11:07:09 CEST         Success
Update lvm.conf file                     node1                 September 24, 2026 11:07:09 CEST         September 24, 2026 11:07:09 CEST         Success
Generating and saving BOM                node0                 September 24, 2026 11:07:10 CEST         September 24, 2026 11:07:39 CEST         Success
Generating and saving BOM                node1                 September 24, 2026 11:07:10 CEST         September 24, 2026 11:07:37 CEST         Success
Update System full patch version         node0                 September 24, 2026 11:07:10 CEST         September 24, 2026 11:07:10 CEST         Success
Update System full patch version         node1                 September 24, 2026 11:07:10 CEST         September 24, 2026 11:07:10 CEST         Success
Update System rebootless patch version   node0                 September 24, 2026 11:07:10 CEST         September 24, 2026 11:07:10 CEST         Success
Update System rebootless patch version   node1                 September 24, 2026 11:07:10 CEST         September 24, 2026 11:07:10 CEST         Success
PreRebootNode Actions                    node0                 September 24, 2026 11:07:39 CEST         September 24, 2026 11:08:53 CEST         Success
PreRebootNode Actions                    node1                 September 24, 2026 11:08:53 CEST         September 24, 2026 11:10:03 CEST         Success
Reboot Node                              node1                 September 24, 2026 11:10:04 CEST         September 24, 2026 11:29:09 CEST         Success
Validate DCS Agent is up                 node1                 September 24, 2026 11:29:08 CEST         September 24, 2026 11:29:08 CEST         Success
Reboot Node                              node0                 September 24, 2026 11:29:10 CEST         September 24, 2026 11:48:15 CEST         Success
Validate DCS Agent is up                 node0                 September 24, 2026 11:48:15 CEST         September 24, 2026 11:48:15 CEST         Success

[root@node0 ~]#

Checking the system, there was no really impact, expect some difference in old kernel in /boot between the two nodes (the synchronization script only run on node0), which was having no impact and would be resolved with next hop patching, and local controller firmware not been updated. The patching process had no space to copy the firmware patches locally on the appropriate directory.

The other patching steps, grid and storage went successfully at well. At the end the components versions were as following:

[root@node0 ~]# odacli describe-component
System Version
--------------
19.30.0.0.0

System Node Name
----------------
node0

Local System Version
--------------------
19.30.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                      19.30.0.0.0          up-to-date

GI                                       19.30.0.0.260120     up-to-date

DB {
     OraDB19000_home3                    19.26.0.0.250121     19.30.0.0.260120
     [cdbx9]
}

DCSCONTROLLER                            19.30.0.0.0          up-to-date

DCSCLI                                   19.30.0.0.0          up-to-date

DCSAGENT                                 19.30.0.0.0          up-to-date

DCSADMIN                                 19.30.0.0.0          up-to-date

OS                                       8.10                 up-to-date

ILOM                                     5.1.5.29.r167569     up-to-date

BIOS                                     62150200             up-to-date

LOCAL CONTROLLER FIRMWARE {
     [c3, c4]                            16.35.4030           16.35.8002
}

SHARED CONTROLLER FIRMWARE {
     [c1, c2]                            16.00.08.00          up-to-date
}

FIRMWAREEXPANDER {
     [c1x0, c2x0]                        0310                 up-to-date
}

LOCAL DISK FIRMWARE {
     [c0d0, c0d1]                        D3MU001              up-to-date
}

SHARED DISK FIRMWARE {
     [c1d0, c1d1, c1d2, c1d3, c1d4,      RXG0                 up-to-date
     c1d5, c1d6, c1d7, c1d8, c1d9,
     c1d10, c1d11, c1d12, c1d13,
     c1d14, c1d15, c1d16, c1d17,
     c1d18, c1d19, c1d20, c1d21,
     c1d22, c1d23, c2d0, c2d1, c2d2,
     c2d3, c2d4, c2d5, c2d6, c2d7,
     c2d8, c2d9, c2d10, c2d11, c2d12,
     c2d13, c2d14, c2d15, c2d16,
     c2d17, c2d18, c2d19, c2d20,
     c2d21, c2d22, c2d23]
}

HMP                                      2.4.10.1.600         up-to-date

System Node Name
----------------
node1

Local System Version
--------------------
19.30.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                      19.30.0.0.0          up-to-date

GI                                       19.30.0.0.260120     up-to-date

DB {
     OraDB19000_home3                    19.26.0.0.250121     19.30.0.0.260120
     [cdbx9]
}

DCSCONTROLLER                            19.30.0.0.0          up-to-date

DCSCLI                                   19.30.0.0.0          up-to-date

DCSAGENT                                 19.30.0.0.0          up-to-date

DCSADMIN                                 19.30.0.0.0          up-to-date

OS                                       8.10                 up-to-date

ILOM                                     5.1.5.29.r167569     up-to-date

BIOS                                     62150200             up-to-date

LOCAL CONTROLLER FIRMWARE {
     [c3, c4]                            16.35.8002           up-to-date
}

SHARED CONTROLLER FIRMWARE {
     [c1, c2]                            16.00.08.00          up-to-date
}

FIRMWAREEXPANDER {
     [c1x0, c2x0]                        0310                 up-to-date
}

LOCAL DISK FIRMWARE {
     [c0d0, c0d1]                        D3MU001              up-to-date
}

SHARED DISK FIRMWARE {
     [c1d0, c1d1, c1d2, c1d3, c1d4,      RXG0                 up-to-date
     c1d5, c1d6, c1d7, c1d8, c1d9,
     c1d10, c1d11, c1d12, c1d13,
     c1d14, c1d15, c1d16, c1d17,
     c1d18, c1d19, c1d20, c1d21,
     c1d22, c1d23, c2d0, c2d1, c2d2,
     c2d3, c2d4, c2d5, c2d6, c2d7,
     c2d8, c2d9, c2d10, c2d11, c2d12,
     c2d13, c2d14, c2d15, c2d16,
     c2d17, c2d18, c2d19, c2d20,
     c2d21, c2d22, c2d23]
}

HMP                                      2.4.10.1.600         up-to-date

[root@node0 ~]#

As we can see the local controller firmware is not up to date on node 0 when updated on node1.

Manual patching of local controller firmware

Let’s patch manually the local controller firmware on node0.

Local controller firmware needs to be patched on node0 only:

[root@node0 ~]# odacli describe-component | grep 'LOCAL CONTROLLER FIRMWARE' -A2
LOCAL CONTROLLER FIRMWARE {
     [c3, c4]                            16.35.4030           16.35.8002
}
--
LOCAL CONTROLLER FIRMWARE {
     [c3, c4]                            16.35.8002           up-to-date
}
[root@node0 ~]#

Listing the controller, we can see that we need to patch c3 and c4 Mellanox controllers from 16.35.4030 to 16.35.8002 version for the fiber channel network cards:

[root@node0 ~]# fwupdate list controller

WARNING: Due to strict MMIO memory settings in the running kernel some network controllers may not be accessible.
         See Hardware Management Pack documentation regarding iomem kernel settings required for
         firmware update of these devices.


==================================================
CONTROLLER
==================================================
ID    Type   Manufacturer   Model     Product Name               FW Version       BIOS Version   EFI Version    FCODE Version  Package Version  NVDATA Version    XML Support
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
c0    HDC    Intel          0x2826    0x4873                     -                -              -              -              -                -                 N/A
c1    SAS    LSI Logic      0x0097    ORCL-EXT-SAS3              16.00.08.00      08.37.00.00    18.00.00.00    01.00.68.00    -                14.01.00.13       N/A
c2    SAS    LSI Logic      0x0097    ORCL-EXT-SAS3              16.00.08.00      08.37.00.00    18.00.00.00    01.00.68.00    -                14.01.00.13       N/A
c3    NET    Mellanox       0x1017    Oracle 25GbE Dual-Port SFP 16.35.4030       -              14.29.15       -              -                -                 N/A
c4    NET    Mellanox       0x1017    Oracle 25GbE Dual-Port SFP 16.35.4030       -              14.29.15       -              -                -                 N/A
[root@node0 ~]#

Let’s go in the appropriate mellanox firmware controller directory from the repository and check that version 8002 file is available:

[root@node0 ~]# cd /opt/oracle/oak/pkgrepos/firmwarecontroller/mellanox/0x1017/16.35.8002/

[root@node0 16.35.8002]# ll
total 16392
-rw-r--r--. 1 root root      290 Feb 11  2026 componentmetadata.xml
-rw-r--r--. 1 root root 16777216 Feb 11  2026 fw-ConnectX5-rel-16_35_8002-8201339_Ax_Bx-UEFI-14.29.15.signed.bin
-rw-r--r--. 1 root root     2463 Feb 11  2026 metadata.xml
[root@node0 16.35.8002]#

Let’s update c3 controller:

[root@node0 16.35.8002]# fwupdate update controller -n c3 -x metadata.xml

The following actions will be taken:
==========================================================
ID        Priority Action      Status      Old Firmware Ver.   Proposed Ver.       New Firmware Ver.   System Reboot
---------------------------------------------------------------------------------------------------------------------------
c3        1        Check FW    Success     16.35.4030          16.35.8002          N/A                 N/A
Do you wish to process the above actions? [y/n]? y
Update of c3 to 16.35.8002.
Updating c3: Success

Verifying all priority 1 updates

Execution Summary
==========================================================
ID        Priority Action      Status      Old Firmware Ver.   Proposed Ver.       New Firmware Ver.   System Reboot
---------------------------------------------------------------------------------------------------------------------------
c3        1        Validate    Success     16.35.4030          16.35.8002          16.35.8002          N/A
[root@node0 16.35.8002]#

And c4 controller:

[root@node0 16.35.8002]# fwupdate update controller -n c4 -x metadata.xml

The following actions will be taken:
==========================================================
ID        Priority Action      Status      Old Firmware Ver.   Proposed Ver.       New Firmware Ver.   System Reboot
---------------------------------------------------------------------------------------------------------------------------
c4        1        Check FW    Success     16.35.4030          16.35.8002          N/A                 N/A
Do you wish to process the above actions? [y/n]? y
Update of c4 to 16.35.8002.
Updating c4: Success

Verifying all priority 1 updates

Execution Summary
==========================================================
ID        Priority Action      Status      Old Firmware Ver.   Proposed Ver.       New Firmware Ver.   System Reboot
---------------------------------------------------------------------------------------------------------------------------
c4        1        Validate    Success     16.35.4030          16.35.8002          16.35.8002          N/A

Let’s check and confirm that all is ok, listing the controllers:

[root@node0 16.35.8002]# fwupdate list controller

WARNING: Due to strict MMIO memory settings in the running kernel some network controllers may not be accessible.
         See Hardware Management Pack documentation regarding iomem kernel settings required for
         firmware update of these devices.


==================================================
CONTROLLER
==================================================
ID    Type   Manufacturer   Model     Product Name               FW Version       BIOS Version   EFI Version    FCODE Version  Package Version  NVDATA Version    XML Support
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
c0    HDC    Intel          0x2826    0x4873                     -                -              -              -              -                -                 N/A
c1    SAS    LSI Logic      0x0097    ORCL-EXT-SAS3              16.00.08.00      08.37.00.00    18.00.00.00    01.00.68.00    -                14.01.00.13       N/A
c2    SAS    LSI Logic      0x0097    ORCL-EXT-SAS3              16.00.08.00      08.37.00.00    18.00.00.00    01.00.68.00    -                14.01.00.13       N/A
c3    NET    Mellanox       0x1017    Oracle 25GbE Dual-Port SFP 16.35.8002       -              14.29.15       -              -                -                 N/A
c4    NET    Mellanox       0x1017    Oracle 25GbE Dual-Port SFP 16.35.8002       -              14.29.15       -              -                -                 N/A
[root@node0 16.35.8002]#

We can check and see that now all both nodes local controller firmware are up to date:

[root@node0 16.35.8002]# odacli describe-component | grep 'LOCAL CONTROLLER FIRMWARE' -A2
LOCAL CONTROLLER FIRMWARE {
     [c3, c4]                            16.35.8002           up-to-date
}
--
LOCAL CONTROLLER FIRMWARE {
     [c3, c4]                            16.35.8002           up-to-date
}
[root@node0 16.35.8002]#

To wrap up…

And finally all our ODA components are up to date.

[root@node0 16.35.8002]# odacli describe-component
System Version
--------------
19.30.0.0.0

System Node Name
----------------
node0

Local System Version
--------------------
19.30.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                      19.30.0.0.0          up-to-date

GI                                       19.30.0.0.260120     up-to-date

DB {
     OraDB19000_home3                    19.26.0.0.250121     19.30.0.0.260120
     [cdbx9]
}

DCSCONTROLLER                            19.30.0.0.0          up-to-date

DCSCLI                                   19.30.0.0.0          up-to-date

DCSAGENT                                 19.30.0.0.0          up-to-date

DCSADMIN                                 19.30.0.0.0          up-to-date

OS                                       8.10                 up-to-date

ILOM                                     5.1.5.29.r167569     up-to-date

BIOS                                     62150200             up-to-date

LOCAL CONTROLLER FIRMWARE {
     [c3, c4]                            16.35.8002           up-to-date
}

SHARED CONTROLLER FIRMWARE {
     [c1, c2]                            16.00.08.00          up-to-date
}

FIRMWAREEXPANDER {
     [c1x0, c2x0]                        0310                 up-to-date
}

LOCAL DISK FIRMWARE {
     [c0d0, c0d1]                        D3MU001              up-to-date
}

SHARED DISK FIRMWARE {
     [c1d0, c1d1, c1d2, c1d3, c1d4,      RXG0                 up-to-date
     c1d5, c1d6, c1d7, c1d8, c1d9,
     c1d10, c1d11, c1d12, c1d13,
     c1d14, c1d15, c1d16, c1d17,
     c1d18, c1d19, c1d20, c1d21,
     c1d22, c1d23, c2d0, c2d1, c2d2,
     c2d3, c2d4, c2d5, c2d6, c2d7,
     c2d8, c2d9, c2d10, c2d11, c2d12,
     c2d13, c2d14, c2d15, c2d16,
     c2d17, c2d18, c2d19, c2d20,
     c2d21, c2d22, c2d23]
}

HMP                                      2.4.10.1.600         up-to-date

System Node Name
----------------
node1

Local System Version
--------------------
19.30.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                      19.30.0.0.0          up-to-date

GI                                       19.30.0.0.260120     up-to-date

DB {
     OraDB19000_home3                    19.26.0.0.250121     19.30.0.0.260120
     [cdbx9]
}

DCSCONTROLLER                            19.30.0.0.0          up-to-date

DCSCLI                                   19.30.0.0.0          up-to-date

DCSAGENT                                 19.30.0.0.0          up-to-date

DCSADMIN                                 19.30.0.0.0          up-to-date

OS                                       8.10                 up-to-date

ILOM                                     5.1.5.29.r167569     up-to-date

BIOS                                     62150200             up-to-date

LOCAL CONTROLLER FIRMWARE {
     [c3, c4]                            16.35.8002           up-to-date
}

SHARED CONTROLLER FIRMWARE {
     [c1, c2]                            16.00.08.00          up-to-date
}

FIRMWAREEXPANDER {
     [c1x0, c2x0]                        0310                 up-to-date
}

LOCAL DISK FIRMWARE {
     [c0d0, c0d1]                        D3MU001              up-to-date
}

SHARED DISK FIRMWARE {
     [c1d0, c1d1, c1d2, c1d3, c1d4,      RXG0                 up-to-date
     c1d5, c1d6, c1d7, c1d8, c1d9,
     c1d10, c1d11, c1d12, c1d13,
     c1d14, c1d15, c1d16, c1d17,
     c1d18, c1d19, c1d20, c1d21,
     c1d22, c1d23, c2d0, c2d1, c2d2,
     c2d3, c2d4, c2d5, c2d6, c2d7,
     c2d8, c2d9, c2d10, c2d11, c2d12,
     c2d13, c2d14, c2d15, c2d16,
     c2d17, c2d18, c2d19, c2d20,
     c2d21, c2d22, c2d23]
}

HMP                                      2.4.10.1.600         up-to-date

[root@node0 16.35.8002]#

We could see how easy it is to manually patch local controller firmware on the ODA, which helps to mitigate patching issues impacted by customized script. This is why, knowing we are speaking about appliance, it is always good to avoid the most we can any customization, and if doing some to well document it.