odacli create-database fails on ODA X7-2HA with java.lang.OutOfMemoryError

Today I was onsite at my customer and he told me: I can no longer create databases on my ODA X7-2HA, every time I try to use odacli create-database it fails, please help.

Ok, let’s check what happens, the customer shares the Oracle Homes, he wants to create a 11.2.0.4 database:

[root@robucnoroda020 ~]# odacli list-dbhomes

ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
157bfdf4-4430-4fb1-878e-2fb803ee54bd     OraDB11204_home1     11.2.0.4.180417 (27441052, 27338049)     /u01/app/oracle/product/11.2.0.4/dbhome_1     Configured
2aaba0e6-4482-4c9f-8d98-4a9d72fdb96e     OraDB12102_home1     12.1.0.2.180417 (27338020, 27338029)     /u01/app/oracle/product/12.1.0.2/dbhome_1     Configured
ad2b0d0a-11c1-4a15-b22a-f698496cd606     OraDB12201_home1     12.2.0.1.180417 (27464465, 27674384)     /u01/app/oracle/product/12.2.0.1/dbhome_1     Configured

[root@robucnoroda020 ~]#

Ok we try to create a 11.2.0.4 database:

[root@robucnoroda020 log]# odacli create-database -n FOO -dh 157bfdf4-4430-4fb1-878e-2fb803ee54bd -cs AL32UTF8 -y RAC -r ACFS -m
Password for SYS,SYSTEM and PDB Admin:

Job details
----------------------------------------------------------------
 ID: 1959838e-34a6-419e-94da-08b931a039cc
 Description: Database service creation with db name: FOO
 Status: Created
 Created: December 4, 2018 11:11:26 PM EET
 Message:

Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[root@robucnoroda020 log]#

The job was created successful, we check whats going on:

[root@robucnoroda020 log]# odacli describe-job -i 1959838e-34a6-419e-94da-08b931a039cc

Job details
----------------------------------------------------------------
 ID: 1959838e-34a6-419e-94da-08b931a039cc
 Description: Database service creation with db name: FOO
 Status: Failure
 Created: December 4, 2018 11:11:26 PM EET
 Message: DCS-10001:Internal error encountered: Failed to create the database FOO.

Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Database Service creation December 4, 2018 11:11:26 PM EET December 4, 2018 11:13:01 PM EET Failure
Database Service creation December 4, 2018 11:11:26 PM EET December 4, 2018 11:13:01 PM EET Failure
Setting up ssh equivalance December 4, 2018 11:11:26 PM EET December 4, 2018 11:11:46 PM EET Success
Creating volume dclFOO December 4, 2018 11:11:47 PM EET December 4, 2018 11:12:04 PM EET Success
Creating volume datFOO December 4, 2018 11:12:04 PM EET December 4, 2018 11:12:21 PM EET Success
Creating ACFS filesystem for DATA December 4, 2018 11:12:21 PM EET December 4, 2018 11:12:34 PM EET Success
Database Service creation December 4, 2018 11:12:34 PM EET December 4, 2018 11:13:01 PM EET Failure
Database Creation December 4, 2018 11:12:34 PM EET December 4, 2018 11:13:00 PM EET Failure

[root@robucnoroda020 log]#

Indeed, the job has failed, next we check the DCS log, there we can see the database creation failure:

2018-12-04 23:13:01,209 DEBUG [Database Service creation] [] c.o.d.c.t.r.TaskReportRecorder:  Compile task plan for ServiceJobReport
'{
  "updatedTime" : null,
  "jobId" : "1959838e-34a6-419e-94da-08b931a039cc",
  "status" : "Failure",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : 1543957886185,
  "resourceList" : [ ],
  "description" : "Database service creation with db name: FOO"
}'...

2018-12-04 23:13:01,219 DEBUG [Database Service creation] [] c.o.d.a.t.TaskServiceRequest: Task[id: 1959838e-34a6-419e-94da-08b931a039cc, jobid: 1959838e-34a6-419e-94da-08b931a039cc, TaskName: Database Service creation] call() completed.
2018-12-04 23:13:01,219 INFO [Database Service creation] [] c.o.d.a.t.TaskServiceRequest: Task[id: 1959838e-34a6-419e-94da-08b931a039cc, jobid: 1959838e-34a6-419e-94da-08b931a039cc, TaskName: Database Service creation] completed: Failure

Ok, the log don’t tell us what’s going wrong, but behind the scene odacli create-database uses dbca in the requested ORACLE Home. In the next step we check the the dbca logs:

oracle@robucnoroda020:/u01/app/oracle/cfgtoollogs/dbca/FOO/ [rdbms11204] ll
total 276
-rw-r----- 1 oracle oinstall 275412 Dec  4 23:13 trace.log
oracle@robucnoroda020:/u01/app/oracle/cfgtoollogs/dbca/FOO/ [rdbms11204]

oracle@robucnoroda020:/u01/app/oracle/cfgtoollogs/dbca/FOO/ [rdbms11204] cat trace.log

......

[main] [ 2018-12-04 23:12:57.546 EET ] [InventoryUtil.getHomeName:111]  homeName = OraDB11204_home1
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at oracle.xml.parser.v2.XMLDocument.createNodeFromType(XMLDocument.java:4132)
	at oracle.xml.parser.v2.XMLDocument.createElement(XMLDocument.java:2801)
	at oracle.xml.parser.v2.DocumentBuilder.startElement(DocumentBuilder.java:488)
	at oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1616)
	at oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:456)
	at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:402)
	at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:244)
	at oracle.xml.jaxp.JXDocumentBuilder.parse(JXDocumentBuilder.java:155)
	at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:172)
	at oracle.sysman.oix.oixd.OixdDOMReader.getDocument(OixdDOMReader.java:42)
	at oracle.sysman.oic.oics.OicsCheckPointReader.buildCheckpoint(OicsCheckPointReader.java:75)
	at oracle.sysman.oic.oics.OicsCheckPointSession.<init>(OicsCheckPointSession.java:101)
	at oracle.sysman.oic.oics.OicsCheckPointIndexSession.<init>(OicsCheckPointIndexSession.java:123)
	at oracle.sysman.oic.oics.OicsCheckPointFactory.getIndexSession(OicsCheckPointFactory.java:69)
	at oracle.sysman.assistants.util.CheckpointContext.getCheckPointSession(CheckpointContext.java:256)
	at oracle.sysman.assistants.util.CheckpointContext.getCheckPoint(CheckpointContext.java:245)
	at oracle.sysman.assistants.dbca.backend.Host.cleanup(Host.java:3710)
	at oracle.sysman.assistants.dbca.backend.SilentHost.cleanup(SilentHost.java:585)
	at oracle.sysman.assistants.dbca.Dbca.execute(Dbca.java:145)
	at oracle.sysman.assistants.dbca.Dbca.main(Dbca.java:189)
[Thread-5] [ 2018-12-04 23:13:00.631 EET ] [DbcaCleanupHook.run:44]  Cleanup started
[Thread-5] [ 2018-12-04 23:13:00.631 EET ] [OracleHome.cleanupDBOptionsIntance:1482]  DB Options dummy instance sid=null
[Thread-5] [ 2018-12-04 23:13:00.631 EET ] [DbcaCleanupHook.run:49]  Cleanup ended

Ah, there is a JAVA OutOfMemory Exception, we know this from older times, we have to change the Heap Space for dbca’s Java  engine, let’s change to ORACLE_HOME and check dbca:

oracle@robucnoroda020:/u01/app/oracle/product/11.2.0.4/dbhome_1/bin/ [rdbms11204] grep JRE_OPT dbca
JRE_OPTIONS="${JRE_OPTIONS} -DSET_LAF=${SET_LAF} -Dsun.java2d.font.DisableAlgorithmicStyles=true -Dice.pilots.html4.ignoreNonGenericFonts=true  -DDISPLAY=${DISPLAY} -DJDBC_PROTOCOL=thin -mx128m"
exec $JRE_DIR/bin/java  $JRE_OPTIONS  $DEBUG_STRING -classpath $CLASSPATH oracle.sysman.assistants.dbca.Dbca $ARGUMENTS

In the dbca script, we see that the Java Heap Space is 128MB, we change it to 512MB (and yes create a backup first):

oracle@robucnoroda020:/u01/app/oracle/product/11.2.0.4/dbhome_1/bin/ [rdbms11204] grep JRE_OPT dbca
JRE_OPTIONS="${JRE_OPTIONS} -DSET_LAF=${SET_LAF} -Dsun.java2d.font.DisableAlgorithmicStyles=true -Dice.pilots.html4.ignoreNonGenericFonts=true  -DDISPLAY=${DISPLAY} -DJDBC_PROTOCOL=thin -mx512m"
exec $JRE_DIR/bin/java  $JRE_OPTIONS  $DEBUG_STRING -classpath $CLASSPATH oracle.sysman.assistants.dbca.Dbca $ARGUMENTS

After deleting the failed database we try again to create our database FOO:

[root@robucnoroda020 log]# odacli create-database -n FOO -dh 157bfdf4-4430-4fb1-878e-2fb803ee54bd -cs AL32UTF8 -y RAC -r ACFS -m
Password for SYS,SYSTEM and PDB Admin:

Job details
----------------------------------------------------------------
                     ID:  b289ed58-c29f-4ea8-8aa8-46a5af8ca529
            Description:  Database service creation with db name: FOO
                 Status:  Created
                Created:  December 4, 2018 11:45:04 PM EET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

Let’s check what’s going on:

[root@robucnoroda020 log]# odacli describe-job -i b289ed58-c29f-4ea8-8aa8-46a5af8ca529

Job details
----------------------------------------------------------------
                     ID:  b289ed58-c29f-4ea8-8aa8-46a5af8ca529
            Description:  Database service creation with db name: FOO
                 Status:  Success
                Created:  December 4, 2018 11:45:04 PM EET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Setting up ssh equivalance               December 4, 2018 11:45:05 PM EET    December 4, 2018 11:45:25 PM EET    Success
Creating volume dclFOO                   December 4, 2018 11:45:25 PM EET    December 4, 2018 11:45:42 PM EET    Success
Creating volume datFOO                   December 4, 2018 11:45:42 PM EET    December 4, 2018 11:45:59 PM EET    Success
Creating ACFS filesystem for DATA        December 4, 2018 11:45:59 PM EET    December 4, 2018 11:46:13 PM EET    Success
Database Service creation                December 4, 2018 11:46:13 PM EET    December 4, 2018 11:51:23 PM EET    Success
Database Creation                        December 4, 2018 11:46:13 PM EET    December 4, 2018 11:49:56 PM EET    Success
updating the Database version            December 4, 2018 11:51:21 PM EET    December 4, 2018 11:51:23 PM EET    Success
create Users tablespace                  December 4, 2018 11:51:23 PM EET    December 4, 2018 11:51:25 PM EET    Success

[root@robucnoroda020 log]#

Super, the database was successfully created. It seems that sometimes odacli create-database fails due to dbca memory usage. So also on ODA check your dbca logs, if your database creation wasn’t successful. If you see these Heap Space Exceptions, don’t be afraid to change dbca’s heap memory allocation.