{"id":2157,"date":"2014-07-18T02:24:00","date_gmt":"2014-07-18T00:24:00","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/"},"modified":"2014-07-18T02:24:00","modified_gmt":"2014-07-18T00:24:00","slug":"oracle-em-agent-12c-thread-leak-on-rac","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/","title":{"rendered":"Oracle EM agent 12c thread leak on RAC"},"content":{"rendered":"

By Franck Pachot<\/h2>\n

.
\nIn a previous post about nproc limit<\/a>, I wrote that I had to investigate the nproc limit with the number of threads because my Oracle 12c EM agent was having thousands of threads. This post is a short feedback about this issue and the way I have found the root cause. It concerns the enterprise manager agent 12c on Grid Infrasctructure >= 11.2.0.2<\/p>\n

NLWP<\/h3>\n

The issue was:<\/p>\n

ps -o nlwp,pid,lwp,args -u oracle | sort -n\nNLWP   PID   LWP COMMAND\n   1  8444  8444 oracleOPRODP3 (LOCAL=NO)\n   1  9397  9397 oracleOPRODP3 (LOCAL=NO)\n   1  9542  9542 oracleOPRODP3 (LOCAL=NO)\n   1  9803  9803 \/u00\/app\/oracle\/product\/agent12c\/core\/12.1.0.3.0\/perl\/bin\/perl \/u00\/app\/oracle\/product\/agent12c\/core\/12.1.0.3.0\/bin\/emwd.pl agent \/u00\/app\/oracle\/product\/agent12c\/agent_inst\/sysman\/log\/emagent.nohup\n  19 11966 11966 \/u00\/app\/11.2.0\/grid\/bin\/oraagent.bin\n1114  9963  9963 \/u00\/app\/oracle\/product\/agent12c\/core\/12.1.0.3.0\/jdk\/bin\/java ... emagentSDK.jar oracle.sysman.gcagent.tmmain.TMMain\n<\/code><\/pre>\n
By default ps has only one entry per process, but each processes can have several threads – implemented on linux as light-weight process (LWP). Here, the NLWP column shows that I have 1114 threads for my EM 12c agent – and it was increasing every day until it reached the limit and the node failed (‘Resource temporarily unavailable’).<\/p>\n
The first thing to do is to know what those threads are. The ps entries do not have a lot of information, but I discovered jstack<\/strong> which every java developer should know, I presume. You probably know that java has very verbose (lengthy) stack traces. Jstack was able to show me thousands of them in only one command:<\/p>\n
Jstack<\/h3>\n
$ jstack 9963\n2014-06-03 13:29:04\nFull thread dump Java HotSpot(TM) 64-Bit Server VM (20.14-b01 mixed mode):\n\n\"Attach Listener\" daemon prio=10 tid=0x00007f3368002000 nid=0x4c9b waiting on condition [0x0000000000000000]\n   java.lang.Thread.State: RUNNABLE\n\n\"CRSeOns\" prio=10 tid=0x00007f32c80b6800 nid=0x3863 in Object.wait() [0x00007f31fe11f000]\n   java.lang.Thread.State: TIMED_WAITING (on object monitor)\n\tat java.lang.Object.wait(Native Method)\n\tat oracle.eons.impl.NotificationQueue.internalDequeue(NotificationQueue.java:278)\n\t- locked  (a java.lang.Object)\n\tat oracle.eons.impl.NotificationQueue.dequeue(NotificationQueue.java:255)\n\tat oracle.eons.proxy.impl.client.base.SubscriberImpl.receive(SubscriberImpl.java:98)\n\tat oracle.eons.proxy.impl.client.base.SubscriberImpl.receive(SubscriberImpl.java:79)\n\tat oracle.eons.proxy.impl.client.ProxySubscriber.receive(ProxySubscriber.java:29)\n\tat oracle.sysman.db.receivelet.eons.EonsMetric.beginSubscription(EonsMetric.java:872)\n\tat oracle.sysman.db.receivelet.eons.EonsMetricWlm.run(EonsMetricWlm.java:139)\n\tat oracle.sysman.gcagent.target.interaction.execution.ReceiveletInteractionMgr$3$1.run(ReceiveletInteractionMgr.java:1401)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)\n\tat oracle.sysman.gcagent.util.system.GCAThread$RunnableWrapper.run(GCAThread.java:184)\n\tat java.lang.Thread.run(Thread.java:662)\n...\n<\/code><\/pre>\n
CRSeOns<\/h3>\n
I don’t paste all of them here. We have the ‘main’, we have a few GCs and ‘Gang workers’ which are present in all JVMs and we have a few enterprise manager threads. And what was interesting was that I had thousands of “CRSeOns” that seemed to be increasing.<\/p>\n
Some guesses: I’m on RAC, and I have a ‘ons’ resource and the EM agent tries to subscribe to it. Goggle search returned nothing, and that’s the reason I put that in a blog post now. Then I searched MOS, and bingo, there is a note: Doc ID 1486626.1<\/a>. It has nothing to do with my issue, but has an interesting comment in it:<\/p>\n
In cluster version 11.2.0.2 and higher, the ora.eons resource functionality has been moved to EVM. Because of this the ora.eons resource no longer exists or is controlled by crsctl.<\/p>\n
It also explains how to disable EM agent subscription:<\/p>\n
emctl setproperty agent -name disableEonsRcvlet -value true<\/code><\/pre>\n
I’m in 11.2.0.3 and I have thousands of threads related to a functionality that doesn’t exist anymore. And that leads to some failures in my 4 nodes cluster.<\/p>\n
The solution was simple: disable it.<\/p>\n
For a long time I have seen a lot of memory leaks or CPU usage leaks related to the enterprise manager agent. With this new issue, I discovered a thread leak and I also faced a SR leak when trying to get support for the ‘Resource temporarily unavailable’ error, going back and forth between OS, Database, Cluster and EM support teams…<\/p>\n","protected":false},"excerpt":{"rendered":"
By Franck Pachot . In a previous post about nproc limit, I wrote that I had to investigate the nproc limit with the number of threads because my Oracle 12c EM agent was having thousands of threads. This post is a short feedback about this issue and the way I have found the root cause. […]<\/p>\n","protected":false},"author":27,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[198,59],"tags":[214,96,215,216],"type_dbi":[],"class_list":["post-2157","post","type-post","status-publish","format-standard","hentry","category-database-management","category-oracle","tag-java","tag-oracle","tag-oracle-enterprise-manager","tag-oracle-rac"],"acf":[],"yoast_head":"\nOracle EM agent 12c thread leak on RAC - dbi Blog<\/title>\n<meta name=\"description\" content=\"A thread leak in Oracle Enterprise Manager 12c revealed by jstack.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Oracle EM agent 12c thread leak on RAC\" \/>\n<meta property=\"og:description\" content=\"A thread leak in Oracle Enterprise Manager 12c revealed by jstack.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2014-07-18T00:24:00+00:00\" \/>\n<meta name=\"author\" content=\"Oracle Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Oracle Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/\"},\"author\":{\"name\":\"Oracle Team\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee\"},\"headline\":\"Oracle EM agent 12c thread leak on RAC\",\"datePublished\":\"2014-07-18T00:24:00+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/\"},\"wordCount\":443,\"commentCount\":0,\"keywords\":[\"Java\",\"Oracle\",\"Oracle Enterprise Manager\",\"Oracle RAC\"],\"articleSection\":[\"Database management\",\"Oracle\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/\",\"name\":\"Oracle EM agent 12c thread leak on RAC - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"datePublished\":\"2014-07-18T00:24:00+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee\"},\"description\":\"A thread leak in Oracle Enterprise Manager 12c revealed by jstack.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Oracle EM agent 12c thread leak on RAC\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee\",\"name\":\"Oracle Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g\",\"caption\":\"Oracle Team\"},\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/oracle-team\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Oracle EM agent 12c thread leak on RAC - dbi Blog","description":"A thread leak in Oracle Enterprise Manager 12c revealed by jstack.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/","og_locale":"en_US","og_type":"article","og_title":"Oracle EM agent 12c thread leak on RAC","og_description":"A thread leak in Oracle Enterprise Manager 12c revealed by jstack.","og_url":"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/","og_site_name":"dbi Blog","article_published_time":"2014-07-18T00:24:00+00:00","author":"Oracle Team","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Oracle Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/"},"author":{"name":"Oracle Team","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee"},"headline":"Oracle EM agent 12c thread leak on RAC","datePublished":"2014-07-18T00:24:00+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/"},"wordCount":443,"commentCount":0,"keywords":["Java","Oracle","Oracle Enterprise Manager","Oracle RAC"],"articleSection":["Database management","Oracle"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/","url":"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/","name":"Oracle EM agent 12c thread leak on RAC - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2014-07-18T00:24:00+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee"},"description":"A thread leak in Oracle Enterprise Manager 12c revealed by jstack.","breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/oracle-em-agent-12c-thread-leak-on-rac\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Oracle EM agent 12c thread leak on RAC"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee","name":"Oracle Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g","caption":"Oracle Team"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/oracle-team\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/2157","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/27"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=2157"}],"version-history":[{"count":0,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/2157\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=2157"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=2157"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=2157"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=2157"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}