At that time there was a network routing issue to the preferred NTP. This is why only the 2 other NTP servers are displayed in the ntpq command. This was certainly the root cause of the synchronization problem between both nodes.<\/p>\n
The refid entry will show the current source of synchronization for each peer. In my case the source was 131.188.3.223 which is a NTP server of the Friedrich-Alexander university in Erlangen (Germany). This can be confirmed by doing a simple nslookup from my laptop.<\/p>\n
\nC:Usersmaw>nslookup 131.188.3.223\nServer: UnKnown\nAddress: 192.168.1.254\n\nName: ntp3.rrze.uni-erlangen.de\nAddress: 131.188.3.223\n<\/pre>\nThe character at the left margin will display the synchronization status for the peer.
\nThe character * will highlight which peer is currently selected. It is named the system peer.
\nThe character + will highlight the peers which are designated as been acceptable for synchronization but not currently selected. They are considered as good reference time sources and are survivors of the selection process. They can be potential system peers and are then called candidates.
\nThe character – will highlight a peer that is discarded from the selection. Those other time sources display a time that differs from the survivors’ time. They are called falsetickers.<\/p>\n
st<\/h4>\n
The st entry will display the peer stratum. The stratum will measure the distance between the peer to the NTP source. That would show how many hops\/servers there are until the hardware refclock. In my case, the customer NTP servers are stratum 2 servers. This indicates that they are synchronized directly with the NTP server of the Friedrich-Alexander university which is itself a stratum 1 which has a hardware refclock (stratum 0). The stratum can be seen as a level in the timing hierarchy.<\/p>\n
t<\/h4>\n
The t enty will display the type of the peer.
\nu=unicast
\nm=multicast
\nl=local
\n-=don’t know<\/p>\n
when<\/h4>\n
The when entry will display the time in seconds since the peer was last heard.<\/p>\n
poll<\/h4>\n
The poll entry will display the polling interval in seconds. This is the time in seconds the NTP daemon took to synchronize with the peer. That’s to say the delay in seconds between 2 requests.<\/p>\n
reach<\/h4>\n
The reach entry will display in octal format the status of the server availability. After 8 successful synchronization attempts the value will be 377.<\/p>\n
delay<\/h4>\n
The delay entry will display the latency the request will take from client to server and back again. It describe the round-trip delay of the NTP request. This value is important for the NTP to take the network latency in account and adjust accordingly the timing.<\/p>\n
offset<\/h4>\n
The offset entry will give the time difference between the ODA itself and the peer (the synchronization server).<\/p>\n
jitter<\/h4>\n
The jitter entry will give the dispersion, the magnitude of variance. Each peer has a different amount of jitter. The lower the jitter value is, more accurate will be the peer.<\/p>\n
Resolving and checking the cluster offset issue<\/h3>\n
Now that the NTP issue is resolved, I could restart Oracle cluster. I could also check the nodes was up and running again.<\/p>\n
I first could see that the ODA date has been automatically adjusted by the NTP :<\/p>\n
\n[root@ODA-node1 ~]# date\nWed Sep 16 11:29:31 CEST 2020\n<\/pre>\nI could restart the cluster :<\/p>\n
\n[root@ODA-node0 bin]# cd \/u01\/app\/19.0.0.0\/grid\/bin\/\n\n[root@ODA-node0 bin]# .\/crsctl start cluster\nCRS-2672: Attempting to start 'ora.ctssd' on 'ODA-node0'\nCRS-2676: Start of 'ora.ctssd' on 'ODA-node0' succeeded\nCRS-2672: Attempting to start 'ora.asm' on 'ODA-node0'\nCRS-2672: Attempting to start 'ora.crsd' on 'ODA-node0'\nCRS-2676: Start of 'ora.crsd' on 'ODA-node0' succeeded\nCRS-2676: Start of 'ora.asm' on 'ODA-node0' succeeded\n<\/pre>\nAnd I could check the cluster is up and running :<\/p>\n
\n[root@ODA-node0 bin]# .\/crsctl check crs\nCRS-4638: Oracle High Availability Services is online\nCRS-4537: Cluster Ready Services is online\nCRS-4529: Cluster Synchronization Services is online\nCRS-4533: Event Manager is online\n\n[root@ODA-node0 bin]# .\/crsctl status resource -t\n--------------------------------------------------------------------------------\nName Target State Server State details\n--------------------------------------------------------------------------------\nLocal Resources\n--------------------------------------------------------------------------------\nora.DATA.COMMONSTORE.advm\n ONLINE ONLINE ODA-node0 STABLE\n ONLINE ONLINE ODA-node1 STABLE\nora.LISTENER.lsnr\n ONLINE ONLINE ODA-node0 STABLE\n ONLINE ONLINE ODA-node1 STABLE\nora.chad\n ONLINE ONLINE ODA-node0 STABLE\n ONLINE ONLINE ODA-node1 STABLE\nora.data.commonstore.acfs\n ONLINE ONLINE ODA-node0 mounted on \/opt\/orac\n le\/dcs\/commonstore,S\n TABLE\n ONLINE ONLINE ODA-node1 mounted on \/opt\/orac\n le\/dcs\/commonstore,S\n TABLE\nora.net1.network\n ONLINE ONLINE ODA-node0 STABLE\n ONLINE ONLINE ODA-node1 STABLE\nora.ons\n ONLINE ONLINE ODA-node0 STABLE\n ONLINE ONLINE ODA-node1 STABLE\nora.proxy_advm\n ONLINE ONLINE ODA-node0 STABLE\n ONLINE ONLINE ODA-node1 STABLE\n--------------------------------------------------------------------------------\nCluster Resources\n--------------------------------------------------------------------------------\nora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)\n 1 ONLINE ONLINE ODA-node0 STABLE\n 2 ONLINE ONLINE ODA-node1 STABLE\nora.DATA.dg(ora.asmgroup)\n 1 ONLINE ONLINE ODA-node0 STABLE\n 2 ONLINE ONLINE ODA-node1 STABLE\nora.LISTENER_SCAN1.lsnr\n 1 ONLINE ONLINE ODA-node0 STABLE\nora.LISTENER_SCAN2.lsnr\n 1 ONLINE ONLINE ODA-node1 STABLE\nora.RECO.dg(ora.asmgroup)\n 1 ONLINE ONLINE ODA-node0 STABLE\n 2 OFFLINE OFFLINE STABLE\nora.asm(ora.asmgroup)\n 1 ONLINE ONLINE ODA-node0 Started,STABLE\n 2 ONLINE ONLINE ODA-node1 STABLE\nora.asmnet1.asmnetwork(ora.asmgroup)\n 1 ONLINE ONLINE ODA-node0 STABLE\n 2 ONLINE ONLINE ODA-node1 STABLE\nora.cvu\n 1 ONLINE ONLINE ODA-node1 STABLE\nora.ODA-node0.vip\n 1 ONLINE ONLINE ODA-node0 STABLE\nora.ODA-node1.vip\n 1 ONLINE ONLINE ODA-node1 STABLE\nora.qosmserver\n 1 ONLINE ONLINE ODA-node1 STABLE\nora.scan1.vip\n 1 ONLINE ONLINE ODA-node0 STABLE\nora.scan2.vip\n 1 ONLINE ONLINE ODA-node1 STABLE\n--------------------------------------------------------------------------------\n<\/pre>\nAnd finally, the ASM instance and the ASM listener were up and running again :<\/p>\n
\noracle@ODA-node0:\/home\/oracle\/ [rdbms12102_1] ser\n2020-09-16_11-34-31::dmk-run.pl::CheckListenerFile ::ERROR ==> Couldn't open the listener.ora : \/u01\/app\/oracle\/product\/12.1.0.2\/dbhome_1\/network\/admin\n\nDummy:\n------\nrdbms12102_1 : DUMMY (12.1.0.2\/dbhome_1)\n\n\nDatabase(s):\n------------\n+ASM1 : STARTED (grid)\n\n\nListener(s):\n------------\nASMNET1LSNR_ASM : Up (grid)\nLISTENER : Up (grid)\nLISTENER_SCAN1 : Up (grid)\n<\/pre>\n","protected":false},"excerpt":{"rendered":"I have been recently deploying some new ODA X8-2-HA. After reimaging and creating the appliance, I had to patch the ODA. The day after, checking status, I could realize that my node 0 was not working as expected. I did some interesting troubleshooting that I wanted to share hoping it might be helpful for someone.<\/p>\n","protected":false},"author":48,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[59],"tags":[38,123,79],"type_dbi":[],"class_list":["post-14863","post","type-post","status-publish","format-standard","hentry","category-oracle","tag-cluster","tag-ntp","tag-oda"],"acf":[],"yoast_head":"\n
ODA X8-2-HA cluster issue and NTP configuration - dbi Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"ODA X8-2-HA cluster issue and NTP configuration\" \/>\n<meta property=\"og:description\" content=\"I have been recently deploying some new ODA X8-2-HA. After reimaging and creating the appliance, I had to patch the ODA. The day after, checking status, I could realize that my node 0 was not working as expected. I did some interesting troubleshooting that I wanted to share hoping it might be helpful for someone.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2020-10-13T08:56:00+00:00\" \/>\n<meta name=\"author\" content=\"Marc Wagner\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Marc Wagner\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"14 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/\"},\"author\":{\"name\":\"Marc Wagner\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/225d9884b8467ead9a872823acb14628\"},\"headline\":\"ODA X8-2-HA cluster issue and NTP configuration\",\"datePublished\":\"2020-10-13T08:56:00+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/\"},\"wordCount\":1087,\"commentCount\":0,\"keywords\":[\"Cluster\",\"NTP\",\"ODA\"],\"articleSection\":[\"Oracle\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/\",\"name\":\"ODA X8-2-HA cluster issue and NTP configuration - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"datePublished\":\"2020-10-13T08:56:00+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/225d9884b8467ead9a872823acb14628\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"ODA X8-2-HA cluster issue and NTP configuration\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/225d9884b8467ead9a872823acb14628\",\"name\":\"Marc Wagner\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/a873cc6e7fbdbbcbdbcaf5dbded14ad9a77b2ec2c3e03b4d724ed33d35d5f328?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a873cc6e7fbdbbcbdbcaf5dbded14ad9a77b2ec2c3e03b4d724ed33d35d5f328?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a873cc6e7fbdbbcbdbcaf5dbded14ad9a77b2ec2c3e03b4d724ed33d35d5f328?s=96&d=mm&r=g\",\"caption\":\"Marc Wagner\"},\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/marc-wagner\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"ODA X8-2-HA cluster issue and NTP configuration - dbi Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/","og_locale":"en_US","og_type":"article","og_title":"ODA X8-2-HA cluster issue and NTP configuration","og_description":"I have been recently deploying some new ODA X8-2-HA. After reimaging and creating the appliance, I had to patch the ODA. The day after, checking status, I could realize that my node 0 was not working as expected. I did some interesting troubleshooting that I wanted to share hoping it might be helpful for someone.","og_url":"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/","og_site_name":"dbi Blog","article_published_time":"2020-10-13T08:56:00+00:00","author":"Marc Wagner","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Marc Wagner","Est. reading time":"14 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/"},"author":{"name":"Marc Wagner","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/225d9884b8467ead9a872823acb14628"},"headline":"ODA X8-2-HA cluster issue and NTP configuration","datePublished":"2020-10-13T08:56:00+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/"},"wordCount":1087,"commentCount":0,"keywords":["Cluster","NTP","ODA"],"articleSection":["Oracle"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/","url":"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/","name":"ODA X8-2-HA cluster issue and NTP configuration - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2020-10-13T08:56:00+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/225d9884b8467ead9a872823acb14628"},"breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/oda-x8-2-ha-cluster-issue-and-ntp-configuration\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"ODA X8-2-HA cluster issue and NTP configuration"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/225d9884b8467ead9a872823acb14628","name":"Marc Wagner","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/a873cc6e7fbdbbcbdbcaf5dbded14ad9a77b2ec2c3e03b4d724ed33d35d5f328?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/a873cc6e7fbdbbcbdbcaf5dbded14ad9a77b2ec2c3e03b4d724ed33d35d5f328?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a873cc6e7fbdbbcbdbcaf5dbded14ad9a77b2ec2c3e03b4d724ed33d35d5f328?s=96&d=mm&r=g","caption":"Marc Wagner"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/marc-wagner\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/14863","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/48"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=14863"}],"version-history":[{"count":0,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/14863\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=14863"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=14863"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=14863"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=14863"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}