Why ODA reimaging doesn’t work on the first try?

Introduction

Reimaging an ODA is a good practice for a lot of reasons. To make your ODA cleaner if it’s running for many years and if you patch regularly. To simplify patching because if you’re late you could have to apply multiple intermediate patches to reach the target version. Or simply because you need to change the configuration (for example network configuration) and you want to make things clean, and be sure that future patches will be OK after the changes.

Understand the reimaging process

Actually, reimaging is divided in two operations. The pure reimaging of the nodes with a dedicated ISO file (barely an OS installation). And then the create appliance part. Reimaging means completely reinstall the ODA from scratch, so you can think that everything will be cleaned up. But reimaging is something slightly different. It only means reinstalling the software on the node. For sure, if you have 2 nodes (HA ODAs) you will have to do the pure reimaging on both nodes. Most of the time pure reimaging is working fine. Just after the reimaging you will need to deploy/create the appliance (for all ODAs using 18.3 or later) from the first node only, and this step can be quite stressful.

Typical problem encountered

This example comes from an ODA X6-2S, but it’s quite the same problem on all the ODAs. You just succesfully reimaged the server, and configured straight the network with odacli-firstnet. As soon as your ODA is in the network, you can copy the GI and DB clones needed to create the appliance. Create the appliance stands for configuring the system for Oracle, installing all the Oracle software, configuring ASM, creating a first database and so on. You’ll find these steps in the official documentation. In this case, a few minutes after running the create-appliance, you discover that the appliance creation failed.

 /opt/oracle/dcs/bin/odacli describe-job -i 3f93ad2d-7f0f-4f25-90cb-d3937b1270a9

Job details
----------------------------------------------------------------
                     ID:  3f93ad2d-7f0f-4f25-90cb-d3937b1270a9
            Description:  Provisioning service creation
                 Status:  Failure
                Created:  May 24, 2019 5:08:50 PM CEST
                Message:  DCS-10001:Internal error encountered: Fail to run root scripts : .

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Provisioning service creation            May 24, 2019 5:08:50 PM CEST        May 24, 2019 5:21:47 PM CEST        Failure
Provisioning service creation            May 24, 2019 5:08:50 PM CEST        May 24, 2019 5:21:47 PM CEST        Failure
networks updation                        May 24, 2019 5:08:51 PM CEST        May 24, 2019 5:08:59 PM CEST        Success
updating network                         May 24, 2019 5:08:51 PM CEST        May 24, 2019 5:08:59 PM CEST        Success
Setting up Network                       May 24, 2019 5:08:51 PM CEST        May 24, 2019 5:08:51 PM CEST        Success
OS usergroup 'asmdba'creation            May 24, 2019 5:08:59 PM CEST        May 24, 2019 5:08:59 PM CEST        Success
OS usergroup 'asmoper'creation           May 24, 2019 5:08:59 PM CEST        May 24, 2019 5:08:59 PM CEST        Success
OS usergroup 'asmadmin'creation          May 24, 2019 5:08:59 PM CEST        May 24, 2019 5:08:59 PM CEST        Success
OS usergroup 'dba'creation               May 24, 2019 5:08:59 PM CEST        May 24, 2019 5:08:59 PM CEST        Success
OS usergroup 'dbaoper'creation           May 24, 2019 5:08:59 PM CEST        May 24, 2019 5:08:59 PM CEST        Success
OS usergroup 'oinstall'creation          May 24, 2019 5:08:59 PM CEST        May 24, 2019 5:08:59 PM CEST        Success
OS user 'grid'creation                   May 24, 2019 5:08:59 PM CEST        May 24, 2019 5:08:59 PM CEST        Success
OS user 'oracle'creation                 May 24, 2019 5:08:59 PM CEST        May 24, 2019 5:08:59 PM CEST        Success
SSH equivalance setup                    May 24, 2019 5:08:59 PM CEST        May 24, 2019 5:08:59 PM CEST        Success
Grid home creation                       May 24, 2019 5:09:06 PM CEST        May 24, 2019 5:11:57 PM CEST        Success
Creating GI home directories             May 24, 2019 5:09:06 PM CEST        May 24, 2019 5:09:06 PM CEST        Success
Cloning Gi home                          May 24, 2019 5:09:06 PM CEST        May 24, 2019 5:11:56 PM CEST        Success
Updating GiHome version                  May 24, 2019 5:11:56 PM CEST        May 24, 2019 5:11:57 PM CEST        Success
Storage discovery                        May 24, 2019 5:11:57 PM CEST        May 24, 2019 5:16:33 PM CEST        Success
Grid stack creation                      May 24, 2019 5:16:33 PM CEST        May 24, 2019 5:21:47 PM CEST        Failure
Configuring GI                           May 24, 2019 5:16:33 PM CEST        May 24, 2019 5:18:13 PM CEST        Success
Running GI root scripts                  May 24, 2019 5:18:13 PM CEST        May 24, 2019 5:21:47 PM CEST        Failure

You were thinking that reimaging was a good idea, and now you’re affraid that maybe not… But don’t panic, it’s a normal behavior.

As this error is related to GI stack configuration, you need to look into the GI logs, as you would do on another platform.

vi /u01/app/18.0.0.0/grid/install/root_odadbi03_2019-05-24_17-18-13-479398837.log

...
2019/05/24 17:20:47 CLSRSC-594: Executing installation step 17 of 20: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'odadbi'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'odadbi' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'odadbi'
CRS-2672: Attempting to start 'ora.mdnsd' on 'odadbi'
CRS-2676: Start of 'ora.mdnsd' on 'odadbi' succeeded
CRS-2676: Start of 'ora.evmd' on 'odadbi' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'odadbi'
CRS-2676: Start of 'ora.gpnpd' on 'odadbi' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'odadbi'
CRS-2672: Attempting to start 'ora.gipcd' on 'odadbi'
CRS-2676: Start of 'ora.cssdmonitor' on 'odadbi' succeeded
CRS-2676: Start of 'ora.gipcd' on 'odadbi' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'odadbi'
CRS-2672: Attempting to start 'ora.diskmon' on 'odadbi'
CRS-2676: Start of 'ora.diskmon' on 'odadbi' succeeded
CRS-2676: Start of 'ora.cssd' on 'odadbi' succeeded
Creating SQL script file /tmp/asminit_sql_2019-05-24-17-21-15.sql
cat: /etc/grub.conf: Permission denied

SQL*Plus: Release 18.0.0.0.0 - Production on Fri May 24 17:21:15 2019
Version 18.3.0.0.0

Copyright (c) 1982, 2018, Oracle.  All rights reserved.

Connected to an idle instance.

ASM instance started

Total System Global Area 1136934472 bytes
Fixed Size                  8666696 bytes
Variable Size            1103101952 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted
ASM diskgroups volume enabled
create diskgroup DATA NORMAL REDUNDANCY
*
ERROR at line 1:
ORA-15018: diskgroup cannot be created
ORA-15030: diskgroup name "DATA" is in use by another diskgroup
...

Problem is quite obvious, diskgroup creation failed when creating DATA diskgroup. As a result, your GI configuration is not complete, and there is no way to go further.

Why ASM is in trouble when reimaging?

Actually, reimaging an ODA will erase all the data on the local disks (the 2 disks for the system) but will not erase anything on the disks dedicated to ASM (the data disks), even if you’re using a lite ODA like most of us these days. As a result, data disks are still configured with previous ASM headers and data, leading to the failure. You can avoid this error by thinking of cleaning up the ODA BEFORE the reimaging (please refer to the dedicated procedure for your version/your ODA). In our case, we totally missed to clean up before, so after the first deployment failure, you’ll find a script for purging ASM tagging and data on disks :

/opt/oracle/oak/onecmd/cleanup.pl
INFO: *******************************************************************
INFO: ** Starting process to cleanup provisioned host odadbi03           **
INFO: *******************************************************************
INFO: Default mode being used to cleanup a provisioned system.
INFO: It will change all ASM disk status from MEMBER to FORMER
Do you want to continue (yes/no) : yes
INFO:
Running cleanup will delete Grid User - 'grid' and
INFO: DB user - 'oracle' and also the
INFO: groups 'oinstall,dba,asmadmin,asmoper,asmdba'
INFO: nodes will be rebooted
Do you want to continue (yes/no) : yes
…

After this cleanup, the server will reboot, and you will be able to retry the odacli create-appliance:

/opt/oracle/dcs/bin/odacli describe-job -i "3571b291-be91-4cd4-a133-b52ead24ff61"

Job details
----------------------------------------------------------------
                     ID:  3571b291-be91-4cd4-a133-b52ead24ff61
            Description:  Provisioning service creation
                 Status:  Success
                Created:  May 24, 2019 5:53:44 PM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
networks updation                        May 24, 2019 5:53:45 PM CEST        May 24, 2019 5:53:52 PM CEST        Success
updating network                         May 24, 2019 5:53:45 PM CEST        May 24, 2019 5:53:52 PM CEST        Success
Setting up Network                       May 24, 2019 5:53:45 PM CEST        May 24, 2019 5:53:45 PM CEST        Success
OS usergroup 'asmdba'creation            May 24, 2019 5:53:52 PM CEST        May 24, 2019 5:53:52 PM CEST        Success
OS usergroup 'asmoper'creation           May 24, 2019 5:53:52 PM CEST        May 24, 2019 5:53:52 PM CEST        Success
OS usergroup 'asmadmin'creation          May 24, 2019 5:53:52 PM CEST        May 24, 2019 5:53:52 PM CEST        Success
OS usergroup 'dba'creation               May 24, 2019 5:53:52 PM CEST        May 24, 2019 5:53:52 PM CEST        Success
OS usergroup 'dbaoper'creation           May 24, 2019 5:53:52 PM CEST        May 24, 2019 5:53:52 PM CEST        Success
OS usergroup 'oinstall'creation          May 24, 2019 5:53:52 PM CEST        May 24, 2019 5:53:52 PM CEST        Success
OS user 'grid'creation                   May 24, 2019 5:53:52 PM CEST        May 24, 2019 5:53:52 PM CEST        Success
OS user 'oracle'creation                 May 24, 2019 5:53:52 PM CEST        May 24, 2019 5:53:52 PM CEST        Success
SSH equivalance setup                    May 24, 2019 5:53:53 PM CEST        May 24, 2019 5:53:53 PM CEST        Success
Grid home creation                       May 24, 2019 5:54:00 PM CEST        May 24, 2019 5:57:08 PM CEST        Success
Creating GI home directories             May 24, 2019 5:54:00 PM CEST        May 24, 2019 5:54:00 PM CEST        Success
Cloning Gi home                          May 24, 2019 5:54:00 PM CEST        May 24, 2019 5:57:06 PM CEST        Success
Updating GiHome version                  May 24, 2019 5:57:06 PM CEST        May 24, 2019 5:57:08 PM CEST        Success
Storage discovery                        May 24, 2019 5:57:08 PM CEST        May 24, 2019 6:01:46 PM CEST        Success
Grid stack creation                      May 24, 2019 6:01:46 PM CEST        May 24, 2019 6:18:04 PM CEST        Success
Configuring GI                           May 24, 2019 6:01:46 PM CEST        May 24, 2019 6:03:27 PM CEST        Success
Running GI root scripts                  May 24, 2019 6:03:27 PM CEST        May 24, 2019 6:12:52 PM CEST        Success
Running GI config assistants             May 24, 2019 6:12:53 PM CEST        May 24, 2019 6:14:45 PM CEST        Success
Setting AUDIT SYSLOG LEVEL               May 24, 2019 6:14:53 PM CEST        May 24, 2019 6:14:53 PM CEST        Success
Post cluster OAKD configuration          May 24, 2019 6:18:04 PM CEST        May 24, 2019 6:21:17 PM CEST        Success
Disk group 'RECO'creation                May 24, 2019 6:21:25 PM CEST        May 24, 2019 6:21:35 PM CEST        Success
Volume 'datDBTEST'creation               May 24, 2019 6:21:35 PM CEST        May 24, 2019 6:22:06 PM CEST        Success
Volume 'reco'creation                    May 24, 2019 6:22:06 PM CEST        May 24, 2019 6:22:25 PM CEST        Success
Volume 'commonstore'creation             May 24, 2019 6:22:25 PM CEST        May 24, 2019 6:22:44 PM CEST        Success
ACFS File system 'DATA'creation          May 24, 2019 6:22:44 PM CEST        May 24, 2019 6:22:59 PM CEST        Success
ACFS File system 'RECO'creation          May 24, 2019 6:22:59 PM CEST        May 24, 2019 6:23:15 PM CEST        Success
ACFS File system 'DATA'creation          May 24, 2019 6:23:15 PM CEST        May 24, 2019 6:23:30 PM CEST        Success
Database home creation                   May 24, 2019 6:23:30 PM CEST        May 24, 2019 6:27:02 PM CEST        Success
Validating dbHome available space        May 24, 2019 6:23:30 PM CEST        May 24, 2019 6:23:30 PM CEST        Success
Creating DbHome Directory                May 24, 2019 6:23:30 PM CEST        May 24, 2019 6:23:30 PM CEST        Success
Extract DB clones                        May 24, 2019 6:23:30 PM CEST        May 24, 2019 6:25:08 PM CEST        Success
Clone Db home                            May 24, 2019 6:25:08 PM CEST        May 24, 2019 6:26:47 PM CEST        Success
Enable DB options                        May 24, 2019 6:26:47 PM CEST        May 24, 2019 6:26:56 PM CEST        Success
Run Root DB scripts                      May 24, 2019 6:26:56 PM CEST        May 24, 2019 6:26:56 PM CEST        Success
Provisioning service creation            May 24, 2019 6:27:02 PM CEST        May 24, 2019 6:35:18 PM CEST        Success
Database Creation                        May 24, 2019 6:27:02 PM CEST        May 24, 2019 6:33:23 PM CEST        Success
Change permission for xdb wallet files   May 24, 2019 6:33:23 PM CEST        May 24, 2019 6:33:23 PM CEST        Success
Place SnapshotCtrlFile in sharedLoc      May 24, 2019 6:33:23 PM CEST        May 24, 2019 6:33:25 PM CEST        Success
SqlPatch upgrade                         May 24, 2019 6:34:46 PM CEST        May 24, 2019 6:35:16 PM CEST        Success
updating the Database version            May 24, 2019 6:35:16 PM CEST        May 24, 2019 6:35:18 PM CEST        Success
users tablespace creation                May 24, 2019 6:35:18 PM CEST        May 24, 2019 6:35:20 PM CEST        Success
Install TFA                              May 24, 2019 6:35:20 PM CEST        May 24, 2019 6:39:53 PM CEST        Success

Everything is OK now.

Conclusion

ODA reimaging does not include data disk formating, you are now aware of that. The other thing reimaging is not doing is to patch the bios, firmwares and all the microcodes in your ODA. So just after the create-appliance, don’t forget to apply the patch even you’re already in the target version.