Introduction
Reimaging an ODA is a good practice for a lot of reasons. To make your ODA cleaner if it’s running for many years and if you patch regularly. To simplify patching because if you’re late you could have to apply multiple intermediate patches to reach the target version. Or simply because you need to change the configuration (for example network configuration) and you want to make things clean, and be sure that future patches will be OK after the changes.
Understand the reimaging process
Actually, reimaging is divided in two operations. The pure reimaging of the nodes with a dedicated ISO file (barely an OS installation). And then the create appliance part. Reimaging means completely reinstall the ODA from scratch, so you can think that everything will be cleaned up. But reimaging is something slightly different. It only means reinstalling the software on the node. For sure, if you have 2 nodes (HA ODAs) you will have to do the pure reimaging on both nodes. Most of the time pure reimaging is working fine. Just after the reimaging you will need to deploy/create the appliance (for all ODAs using 18.3 or later) from the first node only, and this step can be quite stressful.
Typical problem encountered
This example comes from an ODA X6-2S, but it’s quite the same problem on all the ODAs. You just succesfully reimaged the server, and configured straight the network with odacli-firstnet. As soon as your ODA is in the network, you can copy the GI and DB clones needed to create the appliance. Create the appliance stands for configuring the system for Oracle, installing all the Oracle software, configuring ASM, creating a first database and so on. You’ll find these steps in the official documentation. In this case, a few minutes after running the create-appliance, you discover that the appliance creation failed.
/opt/oracle/dcs/bin/odacli describe-job -i 3f93ad2d-7f0f-4f25-90cb-d3937b1270a9
Job details
----------------------------------------------------------------
ID: 3f93ad2d-7f0f-4f25-90cb-d3937b1270a9
Description: Provisioning service creation
Status: Failure
Created: May 24, 2019 5:08:50 PM CEST
Message: DCS-10001:Internal error encountered: Fail to run root scripts : .
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Provisioning service creation May 24, 2019 5:08:50 PM CEST May 24, 2019 5:21:47 PM CEST Failure
Provisioning service creation May 24, 2019 5:08:50 PM CEST May 24, 2019 5:21:47 PM CEST Failure
networks updation May 24, 2019 5:08:51 PM CEST May 24, 2019 5:08:59 PM CEST Success
updating network May 24, 2019 5:08:51 PM CEST May 24, 2019 5:08:59 PM CEST Success
Setting up Network May 24, 2019 5:08:51 PM CEST May 24, 2019 5:08:51 PM CEST Success
OS usergroup 'asmdba'creation May 24, 2019 5:08:59 PM CEST May 24, 2019 5:08:59 PM CEST Success
OS usergroup 'asmoper'creation May 24, 2019 5:08:59 PM CEST May 24, 2019 5:08:59 PM CEST Success
OS usergroup 'asmadmin'creation May 24, 2019 5:08:59 PM CEST May 24, 2019 5:08:59 PM CEST Success
OS usergroup 'dba'creation May 24, 2019 5:08:59 PM CEST May 24, 2019 5:08:59 PM CEST Success
OS usergroup 'dbaoper'creation May 24, 2019 5:08:59 PM CEST May 24, 2019 5:08:59 PM CEST Success
OS usergroup 'oinstall'creation May 24, 2019 5:08:59 PM CEST May 24, 2019 5:08:59 PM CEST Success
OS user 'grid'creation May 24, 2019 5:08:59 PM CEST May 24, 2019 5:08:59 PM CEST Success
OS user 'oracle'creation May 24, 2019 5:08:59 PM CEST May 24, 2019 5:08:59 PM CEST Success
SSH equivalance setup May 24, 2019 5:08:59 PM CEST May 24, 2019 5:08:59 PM CEST Success
Grid home creation May 24, 2019 5:09:06 PM CEST May 24, 2019 5:11:57 PM CEST Success
Creating GI home directories May 24, 2019 5:09:06 PM CEST May 24, 2019 5:09:06 PM CEST Success
Cloning Gi home May 24, 2019 5:09:06 PM CEST May 24, 2019 5:11:56 PM CEST Success
Updating GiHome version May 24, 2019 5:11:56 PM CEST May 24, 2019 5:11:57 PM CEST Success
Storage discovery May 24, 2019 5:11:57 PM CEST May 24, 2019 5:16:33 PM CEST Success
Grid stack creation May 24, 2019 5:16:33 PM CEST May 24, 2019 5:21:47 PM CEST Failure
Configuring GI May 24, 2019 5:16:33 PM CEST May 24, 2019 5:18:13 PM CEST Success
Running GI root scripts May 24, 2019 5:18:13 PM CEST May 24, 2019 5:21:47 PM CEST Failure
You were thinking that reimaging was a good idea, and now you’re affraid that maybe not… But don’t panic, it’s a normal behavior.
As this error is related to GI stack configuration, you need to look into the GI logs, as you would do on another platform.
vi /u01/app/18.0.0.0/grid/install/root_odadbi03_2019-05-24_17-18-13-479398837.log ... 2019/05/24 17:20:47 CLSRSC-594: Executing installation step 17 of 20: 'InitConfig'. CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'odadbi' CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'odadbi' has completed CRS-4133: Oracle High Availability Services has been stopped. CRS-4123: Oracle High Availability Services has been started. CRS-2672: Attempting to start 'ora.evmd' on 'odadbi' CRS-2672: Attempting to start 'ora.mdnsd' on 'odadbi' CRS-2676: Start of 'ora.mdnsd' on 'odadbi' succeeded CRS-2676: Start of 'ora.evmd' on 'odadbi' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'odadbi' CRS-2676: Start of 'ora.gpnpd' on 'odadbi' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'odadbi' CRS-2672: Attempting to start 'ora.gipcd' on 'odadbi' CRS-2676: Start of 'ora.cssdmonitor' on 'odadbi' succeeded CRS-2676: Start of 'ora.gipcd' on 'odadbi' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'odadbi' CRS-2672: Attempting to start 'ora.diskmon' on 'odadbi' CRS-2676: Start of 'ora.diskmon' on 'odadbi' succeeded CRS-2676: Start of 'ora.cssd' on 'odadbi' succeeded Creating SQL script file /tmp/asminit_sql_2019-05-24-17-21-15.sql cat: /etc/grub.conf: Permission denied SQL*Plus: Release 18.0.0.0.0 - Production on Fri May 24 17:21:15 2019 Version 18.3.0.0.0 Copyright (c) 1982, 2018, Oracle. All rights reserved. Connected to an idle instance. ASM instance started Total System Global Area 1136934472 bytes Fixed Size 8666696 bytes Variable Size 1103101952 bytes ASM Cache 25165824 bytes ASM diskgroups mounted ASM diskgroups volume enabled create diskgroup DATA NORMAL REDUNDANCY * ERROR at line 1: ORA-15018: diskgroup cannot be created ORA-15030: diskgroup name "DATA" is in use by another diskgroup ...
Problem is quite obvious, diskgroup creation failed when creating DATA diskgroup. As a result, your GI configuration is not complete, and there is no way to go further.
Why ASM is in trouble when reimaging?
Actually, reimaging an ODA will erase all the data on the local disks (the 2 disks for the system) but will not erase anything on the disks dedicated to ASM (the data disks), even if you’re using a lite ODA like most of us these days. As a result, data disks are still configured with previous ASM headers and data, leading to the failure. You can avoid this error by thinking of cleaning up the ODA BEFORE the reimaging (please refer to the dedicated procedure for your version/your ODA). In our case, we totally missed to clean up before, so after the first deployment failure, you’ll find a script for purging ASM tagging and data on disks :
/opt/oracle/oak/onecmd/cleanup.pl INFO: ******************************************************************* INFO: ** Starting process to cleanup provisioned host odadbi03 ** INFO: ******************************************************************* INFO: Default mode being used to cleanup a provisioned system. INFO: It will change all ASM disk status from MEMBER to FORMER Do you want to continue (yes/no) : yes INFO: Running cleanup will delete Grid User - 'grid' and INFO: DB user - 'oracle' and also the INFO: groups 'oinstall,dba,asmadmin,asmoper,asmdba' INFO: nodes will be rebooted Do you want to continue (yes/no) : yes …
After this cleanup, the server will reboot, and you will be able to retry the odacli create-appliance:
/opt/oracle/dcs/bin/odacli describe-job -i "3571b291-be91-4cd4-a133-b52ead24ff61"
Job details
----------------------------------------------------------------
ID: 3571b291-be91-4cd4-a133-b52ead24ff61
Description: Provisioning service creation
Status: Success
Created: May 24, 2019 5:53:44 PM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
networks updation May 24, 2019 5:53:45 PM CEST May 24, 2019 5:53:52 PM CEST Success
updating network May 24, 2019 5:53:45 PM CEST May 24, 2019 5:53:52 PM CEST Success
Setting up Network May 24, 2019 5:53:45 PM CEST May 24, 2019 5:53:45 PM CEST Success
OS usergroup 'asmdba'creation May 24, 2019 5:53:52 PM CEST May 24, 2019 5:53:52 PM CEST Success
OS usergroup 'asmoper'creation May 24, 2019 5:53:52 PM CEST May 24, 2019 5:53:52 PM CEST Success
OS usergroup 'asmadmin'creation May 24, 2019 5:53:52 PM CEST May 24, 2019 5:53:52 PM CEST Success
OS usergroup 'dba'creation May 24, 2019 5:53:52 PM CEST May 24, 2019 5:53:52 PM CEST Success
OS usergroup 'dbaoper'creation May 24, 2019 5:53:52 PM CEST May 24, 2019 5:53:52 PM CEST Success
OS usergroup 'oinstall'creation May 24, 2019 5:53:52 PM CEST May 24, 2019 5:53:52 PM CEST Success
OS user 'grid'creation May 24, 2019 5:53:52 PM CEST May 24, 2019 5:53:52 PM CEST Success
OS user 'oracle'creation May 24, 2019 5:53:52 PM CEST May 24, 2019 5:53:52 PM CEST Success
SSH equivalance setup May 24, 2019 5:53:53 PM CEST May 24, 2019 5:53:53 PM CEST Success
Grid home creation May 24, 2019 5:54:00 PM CEST May 24, 2019 5:57:08 PM CEST Success
Creating GI home directories May 24, 2019 5:54:00 PM CEST May 24, 2019 5:54:00 PM CEST Success
Cloning Gi home May 24, 2019 5:54:00 PM CEST May 24, 2019 5:57:06 PM CEST Success
Updating GiHome version May 24, 2019 5:57:06 PM CEST May 24, 2019 5:57:08 PM CEST Success
Storage discovery May 24, 2019 5:57:08 PM CEST May 24, 2019 6:01:46 PM CEST Success
Grid stack creation May 24, 2019 6:01:46 PM CEST May 24, 2019 6:18:04 PM CEST Success
Configuring GI May 24, 2019 6:01:46 PM CEST May 24, 2019 6:03:27 PM CEST Success
Running GI root scripts May 24, 2019 6:03:27 PM CEST May 24, 2019 6:12:52 PM CEST Success
Running GI config assistants May 24, 2019 6:12:53 PM CEST May 24, 2019 6:14:45 PM CEST Success
Setting AUDIT SYSLOG LEVEL May 24, 2019 6:14:53 PM CEST May 24, 2019 6:14:53 PM CEST Success
Post cluster OAKD configuration May 24, 2019 6:18:04 PM CEST May 24, 2019 6:21:17 PM CEST Success
Disk group 'RECO'creation May 24, 2019 6:21:25 PM CEST May 24, 2019 6:21:35 PM CEST Success
Volume 'datDBTEST'creation May 24, 2019 6:21:35 PM CEST May 24, 2019 6:22:06 PM CEST Success
Volume 'reco'creation May 24, 2019 6:22:06 PM CEST May 24, 2019 6:22:25 PM CEST Success
Volume 'commonstore'creation May 24, 2019 6:22:25 PM CEST May 24, 2019 6:22:44 PM CEST Success
ACFS File system 'DATA'creation May 24, 2019 6:22:44 PM CEST May 24, 2019 6:22:59 PM CEST Success
ACFS File system 'RECO'creation May 24, 2019 6:22:59 PM CEST May 24, 2019 6:23:15 PM CEST Success
ACFS File system 'DATA'creation May 24, 2019 6:23:15 PM CEST May 24, 2019 6:23:30 PM CEST Success
Database home creation May 24, 2019 6:23:30 PM CEST May 24, 2019 6:27:02 PM CEST Success
Validating dbHome available space May 24, 2019 6:23:30 PM CEST May 24, 2019 6:23:30 PM CEST Success
Creating DbHome Directory May 24, 2019 6:23:30 PM CEST May 24, 2019 6:23:30 PM CEST Success
Extract DB clones May 24, 2019 6:23:30 PM CEST May 24, 2019 6:25:08 PM CEST Success
Clone Db home May 24, 2019 6:25:08 PM CEST May 24, 2019 6:26:47 PM CEST Success
Enable DB options May 24, 2019 6:26:47 PM CEST May 24, 2019 6:26:56 PM CEST Success
Run Root DB scripts May 24, 2019 6:26:56 PM CEST May 24, 2019 6:26:56 PM CEST Success
Provisioning service creation May 24, 2019 6:27:02 PM CEST May 24, 2019 6:35:18 PM CEST Success
Database Creation May 24, 2019 6:27:02 PM CEST May 24, 2019 6:33:23 PM CEST Success
Change permission for xdb wallet files May 24, 2019 6:33:23 PM CEST May 24, 2019 6:33:23 PM CEST Success
Place SnapshotCtrlFile in sharedLoc May 24, 2019 6:33:23 PM CEST May 24, 2019 6:33:25 PM CEST Success
SqlPatch upgrade May 24, 2019 6:34:46 PM CEST May 24, 2019 6:35:16 PM CEST Success
updating the Database version May 24, 2019 6:35:16 PM CEST May 24, 2019 6:35:18 PM CEST Success
users tablespace creation May 24, 2019 6:35:18 PM CEST May 24, 2019 6:35:20 PM CEST Success
Install TFA May 24, 2019 6:35:20 PM CEST May 24, 2019 6:39:53 PM CEST Success
Everything is OK now.
Conclusion
ODA reimaging does not include data disk formating, you are now aware of that. The other thing reimaging is not doing is to patch the bios, firmwares and all the microcodes in your ODA. So just after the create-appliance, don’t forget to apply the patch even you’re already in the target version.
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/JDU_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/DWE_web-min-scaled.jpg)