{"id":3069,"date":"2013-05-10T00:18:00","date_gmt":"2013-05-09T22:18:00","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/"},"modified":"2013-05-10T00:18:00","modified_gmt":"2013-05-09T22:18:00","slug":"rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/","title":{"rendered":"Oracle RAC 11.2.0.2: A disturbing loop in a “ohasd” startup script"},"content":{"rendered":"

Last February, I performed an operating system rolling upgrade on a four-nodes RAC cluster (11.2.0.2). I then faced a strange problem when restarting the operating system…<\/p>\n

The first step of the procedure was to stop all Grid Infrastructure and Database services running on the first node as well as to disable Cluster and ASM autostart. The following command is supposed to prevent Oracle High Availability Service (OHAS) to be run at operating system startup:
\n# crsctl disable crs<\/samp>
\nThen, I powered off the server and asked the storage administrator to disable all LUNs attached to this server, including the one containing Oracle binaries (\/u00).<\/p>\n

This step was necessary because the storage bay requires specific drivers which are not shipped with the Linux installation media. Without drivers, all mountpoints attached to LUNs and detected during the upgrade process would have displayed errors.<\/p>\n

However, when starting the server again to check that all mountpoints were disabled, I saw that the startup procedure was blocked to the OHASD service, preventing the server to finish the startup.<\/p>\n

In fact, even if crs autostart is disabled, a deamon called “ohasd” is still running at server startup. Among other things, it checks and\u00a0indefinitively waits for the presence of CRS binaries. No luck, the LUN attached to the mountpoint containg CRS binaries was disabled…<\/p>\n

We can see this check in \/etc\/init.d\/ohasd file:<\/p>\n

# Wait until it is safe to start CRS daemons\n\u00a0 while [ ! -r $CRSCTL ]\n\u00a0 do\n\u00a0\u00a0\u00a0 $LOGMSG \"Waiting for filesystem containing $CRSCTL.\"\n\u00a0\u00a0\u00a0 $SLEEP $DEP_CHECK_WAIT\n\u00a0 done<\/pre>\n
Where $CRSCTL corresponds to \/u00\/app\/11.2.0\/grid\/bin\/crsctl<\/em><\/p>\n
What is crazy is that the loop is performed no matter if autostart is enabled or not. Just after the loop is done, the script checks if autostart is enabled – thanks to the file \/etc\/oracle\/scls_scr\/$MY_HOST\/root\/ohasdstr, which contains “enable” or “disable” depending of the autostart configuration.<\/p>\n
Why do not check if autostart is enabled before looking for CRS binaries? A question that Oracle does not seem to have answered in 11.2.0.3, because we can see that, even if the ohasd startup mecanism was updated, ohasd is still waiting for CRS binaries:<\/p>\n
# Wait until it is safe to start CRS daemons.\n\u00a0 # Wait for 10 minutes for filesystem to mount\n\u00a0 # Print message to syslog and console\n\u00a0 works=true\n\u00a0 for minutes in 10 9 8 7 6 5 4 3 2 1\n\u00a0 do\n\u00a0\u00a0\u00a0 if [ ! -r $CRSCTL ]\n\u00a0\u00a0\u00a0 then\n\u00a0\u00a0\u00a0\u00a0\u00a0 works=false\n\u00a0\u00a0\u00a0\u00a0\u00a0 log_console \"Waiting $minutes minutes for filesystem containing $CRSCTL.\"\n\u00a0\u00a0\u00a0\u00a0\u00a0 $SLEEP $DEP_CHECK_WAIT\n\u00a0\u00a0\u00a0 else\n\u00a0\u00a0\u00a0\u00a0\u00a0 works=true\n\u00a0\u00a0\u00a0\u00a0\u00a0 break\n\u00a0\u00a0\u00a0 fi\n\u00a0 done<\/pre>\n
As you can see, in 11.2.0.3, the server will now finish to start, but will be waiting 10 minutes. A message is displayed in the log startup:<\/p>\n
Apr 30 15:31:29 rac1 logger: Waiting 10 minutes for filesystem containing \/u00\/app\/11.2.0\/grid\/bin\/crsctl.\nApr 30 15:32:29 rac1 logger: Waiting 9 minutes for filesystem containing \/u00\/app\/11.2.0\/grid\/bin\/crsctl.\nApr 30 15:33:29 rac1 logger: Waiting 8 minutes for filesystem containing \/u00\/app\/11.2.0\/grid\/bin\/crsctl.\nApr 30 15:34:29 rac1 logger: Waiting 7 minutes for filesystem containing \/u00\/app\/11.2.0\/grid\/bin\/crsctl.\n[...]<\/pre>\n
At this time, the workaround I found to allow the server to start immediatly is to prevent “ohasd” service to run by renaming or moving the \/etc\/init.d\/ohasd file, or to comment the loop section in order to skip the infinite loop.<\/p>\n
Hopefully, SSH deamon (if enabled) runs before OHAS deamon. It is possible,\u00a0if the startup procedure is blocked, to access the machine through an SSH session in order to apply this workaround and to restart the server.<\/p>\n
I finally added this pre-requisite to the upgrade procedure for the remaining nodes of the cluster.<\/p>\n","protected":false},"excerpt":{"rendered":"
Last February, I performed an operating system rolling upgrade on a four-nodes RAC cluster (11.2.0.2). I then faced a strange problem when restarting the operating system… The first step of the procedure was to stop all Grid Infrastructure and Database services running on the first node as well as to disable Cluster and ASM autostart. […]<\/p>\n","protected":false},"author":27,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[42],"tags":[258,84,17,216],"type_dbi":[],"class_list":["post-3069","post","type-post","status-publish","format-standard","hentry","category-operating-systems","tag-grid","tag-high-availability","tag-oracle-11g","tag-oracle-rac"],"acf":[],"yoast_head":"\nOracle RAC 11.2.0.2: A disturbing loop in a "ohasd" startup script - dbi Blog<\/title>\n<meta name=\"description\" content=\"Last February, I performed an operating system rolling upgrade on a four-nodes RAC cluster (11.2.0.2). I then faced a strange problem when restarting the operating system... The first step of the procedure was to stop all Grid Infrastructure and Database services running on the first node, and to disable Cluster and ASM autostart. The following command is supposed to prevent Oracle High Availability Service (OHAS) to be run at operating system startup:\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Oracle RAC 11.2.0.2: A disturbing loop in a "ohasd" startup script\" \/>\n<meta property=\"og:description\" content=\"Last February, I performed an operating system rolling upgrade on a four-nodes RAC cluster (11.2.0.2). I then faced a strange problem when restarting the operating system... The first step of the procedure was to stop all Grid Infrastructure and Database services running on the first node, and to disable Cluster and ASM autostart. The following command is supposed to prevent Oracle High Availability Service (OHAS) to be run at operating system startup:\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2013-05-09T22:18:00+00:00\" \/>\n<meta name=\"author\" content=\"Oracle Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Oracle Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/\"},\"author\":{\"name\":\"Oracle Team\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee\"},\"headline\":\"Oracle RAC 11.2.0.2: A disturbing loop in a “ohasd” startup script\",\"datePublished\":\"2013-05-09T22:18:00+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/\"},\"wordCount\":465,\"commentCount\":0,\"keywords\":[\"Grid\",\"High availability\",\"Oracle 11g\",\"Oracle RAC\"],\"articleSection\":[\"Operating systems\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/\",\"name\":\"Oracle RAC 11.2.0.2: A disturbing loop in a \\\"ohasd\\\" startup script - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"datePublished\":\"2013-05-09T22:18:00+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee\"},\"description\":\"Last February, I performed an operating system rolling upgrade on a four-nodes RAC cluster (11.2.0.2). I then faced a strange problem when restarting the operating system... The first step of the procedure was to stop all Grid Infrastructure and Database services running on the first node, and to disable Cluster and ASM autostart. The following command is supposed to prevent Oracle High Availability Service (OHAS) to be run at operating system startup:\",\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Oracle RAC 11.2.0.2: A disturbing loop in a “ohasd” startup script\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee\",\"name\":\"Oracle Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g\",\"caption\":\"Oracle Team\"},\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/oracle-team\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Oracle RAC 11.2.0.2: A disturbing loop in a \"ohasd\" startup script - dbi Blog","description":"Last February, I performed an operating system rolling upgrade on a four-nodes RAC cluster (11.2.0.2). I then faced a strange problem when restarting the operating system... The first step of the procedure was to stop all Grid Infrastructure and Database services running on the first node, and to disable Cluster and ASM autostart. The following command is supposed to prevent Oracle High Availability Service (OHAS) to be run at operating system startup:","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/","og_locale":"en_US","og_type":"article","og_title":"Oracle RAC 11.2.0.2: A disturbing loop in a \"ohasd\" startup script","og_description":"Last February, I performed an operating system rolling upgrade on a four-nodes RAC cluster (11.2.0.2). I then faced a strange problem when restarting the operating system... The first step of the procedure was to stop all Grid Infrastructure and Database services running on the first node, and to disable Cluster and ASM autostart. The following command is supposed to prevent Oracle High Availability Service (OHAS) to be run at operating system startup:","og_url":"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/","og_site_name":"dbi Blog","article_published_time":"2013-05-09T22:18:00+00:00","author":"Oracle Team","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Oracle Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/"},"author":{"name":"Oracle Team","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee"},"headline":"Oracle RAC 11.2.0.2: A disturbing loop in a “ohasd” startup script","datePublished":"2013-05-09T22:18:00+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/"},"wordCount":465,"commentCount":0,"keywords":["Grid","High availability","Oracle 11g","Oracle RAC"],"articleSection":["Operating systems"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/","url":"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/","name":"Oracle RAC 11.2.0.2: A disturbing loop in a \"ohasd\" startup script - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2013-05-09T22:18:00+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee"},"description":"Last February, I performed an operating system rolling upgrade on a four-nodes RAC cluster (11.2.0.2). I then faced a strange problem when restarting the operating system... The first step of the procedure was to stop all Grid Infrastructure and Database services running on the first node, and to disable Cluster and ASM autostart. The following command is supposed to prevent Oracle High Availability Service (OHAS) to be run at operating system startup:","breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/rac-11202-a-disturbing-loop-in-a-qohasdq-startup-script\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Oracle RAC 11.2.0.2: A disturbing loop in a “ohasd” startup script"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee","name":"Oracle Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g","caption":"Oracle Team"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/oracle-team\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/3069","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/27"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=3069"}],"version-history":[{"count":0,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/3069\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=3069"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=3069"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=3069"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=3069"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}