Today I was onsite at my customer and he told me: I can no longer create databases on my ODA X7-2HA, every time I try to use odacli create-database it fails, please help.
Ok, let’s check what happens, the customer shares the Oracle Homes, he wants to create a 11.2.0.4 database:
[root@robucnoroda020 ~]# odacli list-dbhomes ID Name DB Version Home Location Status ---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ---------- 157bfdf4-4430-4fb1-878e-2fb803ee54bd OraDB11204_home1 11.2.0.4.180417 (27441052, 27338049) /u01/app/oracle/product/11.2.0.4/dbhome_1 Configured 2aaba0e6-4482-4c9f-8d98-4a9d72fdb96e OraDB12102_home1 12.1.0.2.180417 (27338020, 27338029) /u01/app/oracle/product/12.1.0.2/dbhome_1 Configured ad2b0d0a-11c1-4a15-b22a-f698496cd606 OraDB12201_home1 12.2.0.1.180417 (27464465, 27674384) /u01/app/oracle/product/12.2.0.1/dbhome_1 Configured [root@robucnoroda020 ~]#
Ok we try to create a 11.2.0.4 database:
[root@robucnoroda020 log]# odacli create-database -n FOO -dh 157bfdf4-4430-4fb1-878e-2fb803ee54bd -cs AL32UTF8 -y RAC -r ACFS -m Password for SYS,SYSTEM and PDB Admin: Job details ---------------------------------------------------------------- ID: 1959838e-34a6-419e-94da-08b931a039cc Description: Database service creation with db name: FOO Status: Created Created: December 4, 2018 11:11:26 PM EET Message: Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- [root@robucnoroda020 log]#
The job was created successful, we check whats going on:
[root@robucnoroda020 log]# odacli describe-job -i 1959838e-34a6-419e-94da-08b931a039cc Job details ---------------------------------------------------------------- ID: 1959838e-34a6-419e-94da-08b931a039cc Description: Database service creation with db name: FOO Status: Failure Created: December 4, 2018 11:11:26 PM EET Message: DCS-10001:Internal error encountered: Failed to create the database FOO. Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- Database Service creation December 4, 2018 11:11:26 PM EET December 4, 2018 11:13:01 PM EET Failure Database Service creation December 4, 2018 11:11:26 PM EET December 4, 2018 11:13:01 PM EET Failure Setting up ssh equivalance December 4, 2018 11:11:26 PM EET December 4, 2018 11:11:46 PM EET Success Creating volume dclFOO December 4, 2018 11:11:47 PM EET December 4, 2018 11:12:04 PM EET Success Creating volume datFOO December 4, 2018 11:12:04 PM EET December 4, 2018 11:12:21 PM EET Success Creating ACFS filesystem for DATA December 4, 2018 11:12:21 PM EET December 4, 2018 11:12:34 PM EET Success Database Service creation December 4, 2018 11:12:34 PM EET December 4, 2018 11:13:01 PM EET Failure Database Creation December 4, 2018 11:12:34 PM EET December 4, 2018 11:13:00 PM EET Failure [root@robucnoroda020 log]#
Indeed, the job has failed, next we check the DCS log, there we can see the database creation failure:
2018-12-04 23:13:01,209 DEBUG [Database Service creation] [] c.o.d.c.t.r.TaskReportRecorder: Compile task plan for ServiceJobReport '{ "updatedTime" : null, "jobId" : "1959838e-34a6-419e-94da-08b931a039cc", "status" : "Failure", "message" : null, "reports" : [ ], "createTimestamp" : 1543957886185, "resourceList" : [ ], "description" : "Database service creation with db name: FOO" }'... 2018-12-04 23:13:01,219 DEBUG [Database Service creation] [] c.o.d.a.t.TaskServiceRequest: Task[id: 1959838e-34a6-419e-94da-08b931a039cc, jobid: 1959838e-34a6-419e-94da-08b931a039cc, TaskName: Database Service creation] call() completed. 2018-12-04 23:13:01,219 INFO [Database Service creation] [] c.o.d.a.t.TaskServiceRequest: Task[id: 1959838e-34a6-419e-94da-08b931a039cc, jobid: 1959838e-34a6-419e-94da-08b931a039cc, TaskName: Database Service creation] completed: Failure
Ok, the log don’t tell us what’s going wrong, but behind the scene odacli create-database uses dbca in the requested ORACLE Home. In the next step we check the the dbca logs:
oracle@robucnoroda020:/u01/app/oracle/cfgtoollogs/dbca/FOO/ [rdbms11204] ll total 276 -rw-r----- 1 oracle oinstall 275412 Dec 4 23:13 trace.log oracle@robucnoroda020:/u01/app/oracle/cfgtoollogs/dbca/FOO/ [rdbms11204] oracle@robucnoroda020:/u01/app/oracle/cfgtoollogs/dbca/FOO/ [rdbms11204] cat trace.log ...... [main] [ 2018-12-04 23:12:57.546 EET ] [InventoryUtil.getHomeName:111] homeName = OraDB11204_home1 Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at oracle.xml.parser.v2.XMLDocument.createNodeFromType(XMLDocument.java:4132) at oracle.xml.parser.v2.XMLDocument.createElement(XMLDocument.java:2801) at oracle.xml.parser.v2.DocumentBuilder.startElement(DocumentBuilder.java:488) at oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1616) at oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:456) at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:402) at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:244) at oracle.xml.jaxp.JXDocumentBuilder.parse(JXDocumentBuilder.java:155) at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:172) at oracle.sysman.oix.oixd.OixdDOMReader.getDocument(OixdDOMReader.java:42) at oracle.sysman.oic.oics.OicsCheckPointReader.buildCheckpoint(OicsCheckPointReader.java:75) at oracle.sysman.oic.oics.OicsCheckPointSession.<init>(OicsCheckPointSession.java:101) at oracle.sysman.oic.oics.OicsCheckPointIndexSession.<init>(OicsCheckPointIndexSession.java:123) at oracle.sysman.oic.oics.OicsCheckPointFactory.getIndexSession(OicsCheckPointFactory.java:69) at oracle.sysman.assistants.util.CheckpointContext.getCheckPointSession(CheckpointContext.java:256) at oracle.sysman.assistants.util.CheckpointContext.getCheckPoint(CheckpointContext.java:245) at oracle.sysman.assistants.dbca.backend.Host.cleanup(Host.java:3710) at oracle.sysman.assistants.dbca.backend.SilentHost.cleanup(SilentHost.java:585) at oracle.sysman.assistants.dbca.Dbca.execute(Dbca.java:145) at oracle.sysman.assistants.dbca.Dbca.main(Dbca.java:189) [Thread-5] [ 2018-12-04 23:13:00.631 EET ] [DbcaCleanupHook.run:44] Cleanup started [Thread-5] [ 2018-12-04 23:13:00.631 EET ] [OracleHome.cleanupDBOptionsIntance:1482] DB Options dummy instance sid=null [Thread-5] [ 2018-12-04 23:13:00.631 EET ] [DbcaCleanupHook.run:49] Cleanup ended
Ah, there is a JAVA OutOfMemory Exception, we know this from older times, we have to change the Heap Space for dbca’s Java engine, let’s change to ORACLE_HOME and check dbca:
oracle@robucnoroda020:/u01/app/oracle/product/11.2.0.4/dbhome_1/bin/ [rdbms11204] grep JRE_OPT dbca JRE_OPTIONS="${JRE_OPTIONS} -DSET_LAF=${SET_LAF} -Dsun.java2d.font.DisableAlgorithmicStyles=true -Dice.pilots.html4.ignoreNonGenericFonts=true -DDISPLAY=${DISPLAY} -DJDBC_PROTOCOL=thin -mx128m" exec $JRE_DIR/bin/java $JRE_OPTIONS $DEBUG_STRING -classpath $CLASSPATH oracle.sysman.assistants.dbca.Dbca $ARGUMENTS
In the dbca script, we see that the Java Heap Space is 128MB, we change it to 512MB (and yes create a backup first):
oracle@robucnoroda020:/u01/app/oracle/product/11.2.0.4/dbhome_1/bin/ [rdbms11204] grep JRE_OPT dbca JRE_OPTIONS="${JRE_OPTIONS} -DSET_LAF=${SET_LAF} -Dsun.java2d.font.DisableAlgorithmicStyles=true -Dice.pilots.html4.ignoreNonGenericFonts=true -DDISPLAY=${DISPLAY} -DJDBC_PROTOCOL=thin -mx512m" exec $JRE_DIR/bin/java $JRE_OPTIONS $DEBUG_STRING -classpath $CLASSPATH oracle.sysman.assistants.dbca.Dbca $ARGUMENTS
After deleting the failed database we try again to create our database FOO:
[root@robucnoroda020 log]# odacli create-database -n FOO -dh 157bfdf4-4430-4fb1-878e-2fb803ee54bd -cs AL32UTF8 -y RAC -r ACFS -m Password for SYS,SYSTEM and PDB Admin: Job details ---------------------------------------------------------------- ID: b289ed58-c29f-4ea8-8aa8-46a5af8ca529 Description: Database service creation with db name: FOO Status: Created Created: December 4, 2018 11:45:04 PM EET Message: Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ----------
Let’s check what’s going on:
[root@robucnoroda020 log]# odacli describe-job -i b289ed58-c29f-4ea8-8aa8-46a5af8ca529 Job details ---------------------------------------------------------------- ID: b289ed58-c29f-4ea8-8aa8-46a5af8ca529 Description: Database service creation with db name: FOO Status: Success Created: December 4, 2018 11:45:04 PM EET Message: Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- Setting up ssh equivalance December 4, 2018 11:45:05 PM EET December 4, 2018 11:45:25 PM EET Success Creating volume dclFOO December 4, 2018 11:45:25 PM EET December 4, 2018 11:45:42 PM EET Success Creating volume datFOO December 4, 2018 11:45:42 PM EET December 4, 2018 11:45:59 PM EET Success Creating ACFS filesystem for DATA December 4, 2018 11:45:59 PM EET December 4, 2018 11:46:13 PM EET Success Database Service creation December 4, 2018 11:46:13 PM EET December 4, 2018 11:51:23 PM EET Success Database Creation December 4, 2018 11:46:13 PM EET December 4, 2018 11:49:56 PM EET Success updating the Database version December 4, 2018 11:51:21 PM EET December 4, 2018 11:51:23 PM EET Success create Users tablespace December 4, 2018 11:51:23 PM EET December 4, 2018 11:51:25 PM EET Success [root@robucnoroda020 log]#
Super, the database was successfully created. It seems that sometimes odacli create-database fails due to dbca memory usage. So also on ODA check your dbca logs, if your database creation wasn’t successful. If you see these Heap Space Exceptions, don’t be afraid to change dbca’s heap memory allocation.