{"id":10889,"date":"2018-02-17T13:40:35","date_gmt":"2018-02-17T12:40:35","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\/"},"modified":"2018-02-17T13:40:35","modified_gmt":"2018-02-17T12:40:35","slug":"what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\/","title":{"rendered":"What you can do when your Veritas cluster shows interfaces as down"},"content":{"rendered":"<p>Recently we had the situation that the Veritas cluster (InfoScale 7.3) showed interfaces as down on the two RedHat 7.3 nodes. This e.g. can happen when you change hardware. Although all service groups were up and running this is a situation you usually want to avoid as you never know what happens when the cluster is in such a state. When you have something like this:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\n[root@xxxxx-node1 ~]$ lltstat -nvv | head\nLLT node information:\nNode State Link Status Address\n  * 0 xxxxx-node1 OPEN\n      eth3 UP yy:yy:yy:yy:yy:yy\n      eth1 UP xx:xx:xx:xx:xx:xx\n      bond0 UP rr:rr:rr:rr:rr:rr\n    1 xxxxx-node2 OPEN\n      eth3 UP ee:ee:ee:ee:ee:ee\n      eth1 DOWN tt:tt:tt:tt:tt:tt\n      bond0 DOWN qq:qq:qq:qq:qq:qq\n<\/pre>\n<p>&#8230; what can you do?<\/p>\n<p><!--more--><\/p>\n<p>In our configuration eth1 and eth3 are used for the interconnect and bond0 is the public network. As you can see above the eth1 and bond0 are reported as down for the second node. Of course, the first check you need to do is to check the interface status on the operating system level, but that was fine in our case.<\/p>\n<p>Veritas comes with a tiny little utility (dlpiping) you can use to check connectivity on the Veritas level. Using the information from the lltstat command you can start dlpiping in &#8220;send&#8221; mode on the first node:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\n[root@xxxxx-node1 ~]$ \/opt\/VRTSllt\/dlpiping -vs eth1\n<\/pre>\n<p>When that is running (will not detach from the terminal) you should start in &#8220;receive&#8221; mode on the second node:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\n[root@xxxxx-node1 ~]$ \/opt\/VRTSllt\/dlpiping -vc eth1 xx:xx:xx:xx:xx:xx\nusing packet size = 78\ndlpiping: sent a request to xx:xx:xx:xx:xx:xx\ndlpiping: received a packet from xx:xx:xx:xx:xx:xx\n<\/pre>\n<p>This confirms that connectivity is fine for eth1. When you repeat that for the remaining interfaces (eth3 and bond0) and all is fine then you you can proceed. If not, then you have another issue than what we faced.<\/p>\n<p>The next step is to freeze all your service groups so the cluster will not touch them:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\n[root@xxxxx-node1 ~]$ haconf -makerw\n[root@xxxxx-node1 ~]$ hagrp -freeze SERVICE_GROUP -persistent # do that for all service groups you have defined in the cluster\n[root@xxxxx-node1 ~]$ haconf -dump -makerw\n<\/pre>\n<p>Now the magic:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\n[root@xxxxx-node1 ~]$ hastop -all -force \n<\/pre>\n<p>Why magic? This command will stop the cluster stack on all nodes BUT it will leave all the resources running. So you can do that without shutting down any user defined cluster services (Oracle databases in our case). Once the stack is down on all the nodes stop gab and ltt on both nodes as well:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\n[root@xxxxx-node1 ~]$ systemctl stop gab\n[root@xxxxx-node1 ~]$ systemctl stop llt\n<\/pre>\n<p>Having stopped llt and gab you just need to start them again in the correct order on both systems:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\n[root@xxxxx-node1 ~]$ systemctl start llt\n[root@xxxxx-node1 ~]$ systemctl start gab\n<\/pre>\n<p>&#8230; and after that start the cluster:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\n[root@xxxxx-node1 ~]$ systemctl start vcs\n<\/pre>\n<p>In our case that was enough to make llt work as expected again and the cluster is fine:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\n[root@xxxxx-node1 ~]$ gabconfig -a\nGAB Port Memberships\n===============================================================\nPort a gen f44203 membership 01\nPort h gen f44204 membership 01\n[root@xxxxx-node1 ~]#\n\n[root@xxxxx-node1 ~]$ lltstat -nvv | head\nLLT node information:\n   Node State Link Status Address\n    * 0 xxxxx-node1 OPEN\n      eth3 UP yy:yy:yy:yy:yy:yy\n      eth1 UP xx:xx:xx:xx:xx:xx\n      bond0 UP rr:rr:rr:rr:rr:rr\n    1 xxxxx-node2 OPEN\n      eth3 UP ee:ee:ee:ee:ee:ee\n      eth1 UP qq:qq:qq:qq:qq:qq\n      bond0 UP tt:tt:tt:tt:tt:tt \n<\/pre>\n<p>Hope that helps &#8230;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Recently we had the situation that the Veritas cluster (InfoScale 7.3) showed interfaces as down on the two RedHat 7.3 nodes. This e.g. can happen when you change hardware. Although all service groups were up and running this is a situation you usually want to avoid as you never know what happens when the cluster [&hellip;]<\/p>\n","protected":false},"author":29,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[229],"tags":[1297,96,1223],"type_dbi":[],"class_list":["post-10889","post","type-post","status-publish","format-standard","hentry","category-database-administration-monitoring","tag-infoscale","tag-oracle","tag-veritas"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.2 (Yoast SEO v27.5) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>What you can do when your Veritas cluster shows interfaces as down - dbi Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What you can do when your Veritas cluster shows interfaces as down\" \/>\n<meta property=\"og:description\" content=\"Recently we had the situation that the Veritas cluster (InfoScale 7.3) showed interfaces as down on the two RedHat 7.3 nodes. This e.g. can happen when you change hardware. Although all service groups were up and running this is a situation you usually want to avoid as you never know what happens when the cluster [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2018-02-17T12:40:35+00:00\" \/>\n<meta name=\"author\" content=\"Daniel Westermann\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@westermanndanie\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Daniel Westermann\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\\\/\"},\"author\":{\"name\":\"Daniel Westermann\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/8d08e9bd996a89bd75c0286cbabf3c66\"},\"headline\":\"What you can do when your Veritas cluster shows interfaces as down\",\"datePublished\":\"2018-02-17T12:40:35+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\\\/\"},\"wordCount\":359,\"commentCount\":0,\"keywords\":[\"InfoScale\",\"Oracle\",\"Veritas\"],\"articleSection\":[\"Database Administration &amp; Monitoring\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\\\/\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\\\/\",\"name\":\"What you can do when your Veritas cluster shows interfaces as down - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#website\"},\"datePublished\":\"2018-02-17T12:40:35+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/8d08e9bd996a89bd75c0286cbabf3c66\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What you can do when your Veritas cluster shows interfaces as down\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/8d08e9bd996a89bd75c0286cbabf3c66\",\"name\":\"Daniel Westermann\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g\",\"caption\":\"Daniel Westermann\"},\"description\":\"Daniel Westermann is Principal Consultant and Technology Leader Open Infrastructure at dbi services. He has more than 15 years of experience in management, engineering and optimization of databases and infrastructures, especially on Oracle and PostgreSQL. Since the beginning of his career, he has specialized in Oracle Technologies and is Oracle Certified Professional 12c and Oracle Certified Expert RAC\\\/GridInfra. Over time, Daniel has become increasingly interested in open source technologies, becoming \u201cTechnology Leader Open Infrastructure\u201d and PostgreSQL expert. \u00a0Based on community or EnterpriseDB tools, he develops and installs complex high available solutions with PostgreSQL. He is also a certified PostgreSQL Plus 9.0 Professional and a Postgres Advanced Server 9.4 Professional. He is a regular speaker at PostgreSQL conferences in Switzerland and Europe. Today Daniel is also supporting our customers on AWS services such as AWS RDS, database migrations into the cloud, EC2 and automated infrastructure management with AWS SSM (System Manager). He is a certified AWS Solutions Architect Professional. Prior to dbi services, Daniel was Management System Engineer at LC SYSTEMS-Engineering AG in Basel. Before that, he worked as Oracle Developper &amp;\u00a0Project Manager at Delta Energy Solutions AG in Basel (today Powel AG). Daniel holds a diploma in Business Informatics (DHBW, Germany). His branch-related experience mainly covers the pharma industry, the financial sector, energy, lottery and telecommunications.\",\"sameAs\":[\"https:\\\/\\\/x.com\\\/westermanndanie\"],\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/author\\\/daniel-westermann\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"What you can do when your Veritas cluster shows interfaces as down - dbi Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\/","og_locale":"en_US","og_type":"article","og_title":"What you can do when your Veritas cluster shows interfaces as down","og_description":"Recently we had the situation that the Veritas cluster (InfoScale 7.3) showed interfaces as down on the two RedHat 7.3 nodes. This e.g. can happen when you change hardware. Although all service groups were up and running this is a situation you usually want to avoid as you never know what happens when the cluster [&hellip;]","og_url":"https:\/\/www.dbi-services.com\/blog\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\/","og_site_name":"dbi Blog","article_published_time":"2018-02-17T12:40:35+00:00","author":"Daniel Westermann","twitter_card":"summary_large_image","twitter_creator":"@westermanndanie","twitter_misc":{"Written by":"Daniel Westermann","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\/"},"author":{"name":"Daniel Westermann","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66"},"headline":"What you can do when your Veritas cluster shows interfaces as down","datePublished":"2018-02-17T12:40:35+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\/"},"wordCount":359,"commentCount":0,"keywords":["InfoScale","Oracle","Veritas"],"articleSection":["Database Administration &amp; Monitoring"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\/","url":"https:\/\/www.dbi-services.com\/blog\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\/","name":"What you can do when your Veritas cluster shows interfaces as down - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2018-02-17T12:40:35+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66"},"breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/what-you-can-do-when-your-veritas-cluster-shows-interfaces-as-down\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"What you can do when your Veritas cluster shows interfaces as down"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d08e9bd996a89bd75c0286cbabf3c66","name":"Daniel Westermann","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/31350ceeecb1dd8986339a29bf040d4cd3cd087d410deccd8f55234466d6c317?s=96&d=mm&r=g","caption":"Daniel Westermann"},"description":"Daniel Westermann is Principal Consultant and Technology Leader Open Infrastructure at dbi services. He has more than 15 years of experience in management, engineering and optimization of databases and infrastructures, especially on Oracle and PostgreSQL. Since the beginning of his career, he has specialized in Oracle Technologies and is Oracle Certified Professional 12c and Oracle Certified Expert RAC\/GridInfra. Over time, Daniel has become increasingly interested in open source technologies, becoming \u201cTechnology Leader Open Infrastructure\u201d and PostgreSQL expert. \u00a0Based on community or EnterpriseDB tools, he develops and installs complex high available solutions with PostgreSQL. He is also a certified PostgreSQL Plus 9.0 Professional and a Postgres Advanced Server 9.4 Professional. He is a regular speaker at PostgreSQL conferences in Switzerland and Europe. Today Daniel is also supporting our customers on AWS services such as AWS RDS, database migrations into the cloud, EC2 and automated infrastructure management with AWS SSM (System Manager). He is a certified AWS Solutions Architect Professional. Prior to dbi services, Daniel was Management System Engineer at LC SYSTEMS-Engineering AG in Basel. Before that, he worked as Oracle Developper &amp;\u00a0Project Manager at Delta Energy Solutions AG in Basel (today Powel AG). Daniel holds a diploma in Business Informatics (DHBW, Germany). His branch-related experience mainly covers the pharma industry, the financial sector, energy, lottery and telecommunications.","sameAs":["https:\/\/x.com\/westermanndanie"],"url":"https:\/\/www.dbi-services.com\/blog\/author\/daniel-westermann\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/10889","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/29"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=10889"}],"version-history":[{"count":0,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/10889\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=10889"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=10889"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=10889"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=10889"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}