{"id":3836,"date":"2014-06-10T05:14:00","date_gmt":"2014-06-10T03:14:00","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/"},"modified":"2014-06-10T05:14:00","modified_gmt":"2014-06-10T03:14:00","slug":"linux-how-to-monitor-the-nproc-limit-1","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/","title":{"rendered":"Linux: how to monitor the nproc limit"},"content":{"rendered":"

By Franck Pachot<\/h2>\n

.
\nYou probably know about ‘nproc’ limits in Linux which are set in \/etc\/limits.conf and checked with ‘ulimit -u’. But do you know how to handle the monitoring and be alerted when you’re close the fixed limit?<\/p>\n

Nproc and ps<\/h3>\n

Nproc is defined at OS level to limit the number of processes per user. Oracle 11.2.0.4 documentation recommends the following:<\/p>\n

oracle soft nproc 2047\noracle hard nproc 16384\n<\/code><\/pre>\n
But that is often too low, especially when you have the Enterprise Manager agent or other java programs running.<\/p>\n
Do you want to check that you are far from the limit? then you can use ‘ps’. But beware, ‘ps’ by default does not show all processes.
\nIn Linux, when doing multithreading, each thread is implemented as a light-weight process (LWP). And you must use the ‘-L’ to see all of them.<\/p>\n
Let’s take an example. I have a system where ‘ps -u oracle’ returns 243 lines. But including LWPs shows a lot more processes which is near the limit:<\/p>\n
$ ps h -Led -o user | sort | uniq -c | sort -n\n      1 dbus\n      1 ntp\n      1 rpc\n      1 rpcuser\n      2 avahi\n      2 haldaemon\n      2 postfix\n    166 grid\n    400 root\n   1370 oracle\n<\/code><\/pre>\n
So the ‘oracle’ user has 1370 processes. That’s high. And this is the actual number where the nproc limit applies.<\/p>\n
‘ps -Lf’ can show the detail. And even without ‘-L’ we can display the NLWP which is the number of threads per process:<\/p>\n
ps -o nlwp,pid,lwp,args -u oracle | sort -n\nNLWP   PID   LWP COMMAND\n   1  8444  8444 oracleOPRODP3 (LOCAL=NO)\n   1  9397  9397 oracleOPRODP3 (LOCAL=NO)\n   1  9542  9542 oracleOPRODP3 (LOCAL=NO)\n   1  9803  9803 \/u00\/app\/oracle\/product\/agent12c\/core\/12.1.0.3.0\/perl\/bin\/perl \/u00\/app\/oracle\/product\/agent12c\/core\/12.1.0.3.0\/bin\/emwd.pl agent \/u00\/app\/oracle\/product\/agent12c\/agent_inst\/sysman\/log\/emagent.nohup\n  19 11966 11966 \/u00\/app\/11.2.0\/grid\/bin\/oraagent.bin\n1114  9963  9963 \/u00\/app\/oracle\/product\/agent12c\/core\/12.1.0.3.0\/jdk\/bin\/java ... emagentSDK.jar oracle.sysman.gcagent.tmmain.TMMain\n<\/code><\/pre>\n
The Oracle 12c EM agent has started 1114 threads and the grid infrastructure ‘oraagent.bin’ has 19 threads. In addition to that I’ve a lot of other monothreaded processes. This is how we reach 1370 which is the exact value to compare to the nproc limit.<\/p>\n
So what are the good values to set? About the high number of threads for EM agent 12c, there are a few bugs. And I suspect that 1000 threads is too much, especially when checking them with ‘jstack’ I see that they are “CRSeOns” threads that should not be used in 11.2.0.2 and higher. But that’s another problem which I’m currently investigating. When you reach the nproc limit, the user will not be able to create new processes. clone() calls will return EAGAIN and that is reported by Oracle as:<\/p>\n
ORA-27300: OS system dependent operation:fork failed with status: 11 \nORA-27301: OS failure message: Resource temporarily unavailable \n<\/code><\/pre>\n
And that is clearly bad when it concerns an +ASM instance or archiver processes.<\/p>\n
The goal of the nproc limit is only to prevent ‘fork bombs’ where a process forks forever and exhausts all resources. So there is no problem to increase this limit. However if you set it high for some users (‘oracle’ and ‘grid’ usually), it can be a good idea to monitor the number of processes with the ps h -L above. Because having too many processes is suspect and increasing the limit just hides a process leak and defer the failure.<\/p>\n
In ‘ps h -L -o’ The argument ‘h’ is there to remove the header line, and ‘-L’ to show all processes including LWP. Then you can count with ‘wc -l’.<\/p>\n
The proof<\/h3>\n
In order to be sure that ‘ps h -L’ gives the exact number, I have tested it. In case you want to check this on your system, here is how to do it. And please report any difference.<\/p>\n
First, set your limit to 1024 processes. This is a limit for my user, and the limit is set for my shell and all its child processes:<\/p>\n
[oracle@VM211 ocm]$ ulimit -u 1024\n<\/code><\/pre>\n
Now you can check it:<\/p>\n
[oracle@VM211 ocm]$ ulimit -a\ncore file size          (blocks, -c) 0\ndata seg size           (kbytes, -d) unlimited\nscheduling priority             (-e) 0\nfile size               (blocks, -f) unlimited\npending signals                 (-i) 15919\nmax locked memory       (kbytes, -l) 64\nmax memory size         (kbytes, -m) unlimited\nopen files                      (-n) 1024\npipe size            (512 bytes, -p) 8\nPOSIX message queues     (bytes, -q) 819200\nreal-time priority              (-r) 0\nstack size              (kbytes, -s) 10240\ncpu time               (seconds, -t) unlimited\nmax user processes (-u) 1024\nvirtual memory          (kbytes, -v) unlimited\nfile locks                      (-x) unlimited\n<\/code><\/pre>\n
Then you can run a small C program (testnproc.zip<\/a>) that calls fork() in a loop until it fails with EAGAIN:<\/p>\n
[oracle@VM211 ocm]$ .\/testnproc\n...\nparent says fork number 871 sucessful\n child says fork number 872 pid 1518\nparent says fork number 872 sucessful\n child says fork number 873 pid 1519\nparent says fork number 873 sucessful\n child says fork number 874 pid 1520\nparent says fork number 874 sucessful\nparent says fork number 875 failed (nproc: soft=1024 hard=1024) with errno=11\n<\/code><\/pre>\n
And finally, because the processes sleep for a while, you can check how many processes you have. I do that from another user account for the simple reason that I need to create 2 more processes (‘ps’ and ‘wc’) for that:<\/p>\n
[root@VM211 ocm]# ps h -Lu oracle | wc -l\n1023\n<\/code><\/pre>\n
Recommended values for Oracle<\/h3>\n
Currently this is what is set on Oracle linux 6 for 11gR2 by the preinstall package (in \/etc\/security\/limits.conf):<\/p>\n
oracle   soft   nproc    16384\noracle   hard   nproc    16384\n<\/code><\/pre>\n
For 12c, these are set in \/etc\/security\/limits.d\/oracle-rdbms-server-12cR1-preinstall.conf which overrides \/etc\/security\/limits.conf:<\/p>\n
oracle soft nproc 16384\noracle hard nproc 16384\n<\/code><\/pre>\n
And just for your information, here is what is set in the ODA X4-2:<\/p>\n
oracle soft nproc 131072\n<\/code><\/pre>\n
So what do you want to set? You probably don’t want it too low and experience ‘resource temporarily unavailable’. But what you don’t want either is 100000 processes on your server. So my recommendation is to set it high but monitor it when the number of processes reaches something that is not sensible. Then you prevent having the system down in case of process leak, but you can detect it and ask for a patch.<\/p>\n","protected":false},"excerpt":{"rendered":"
By Franck Pachot . You probably know about ‘nproc’ limits in Linux which are set in \/etc\/limits.conf and checked with ‘ulimit -u’. But do you know how to handle the monitoring and be alerted when you’re close the fixed limit? Nproc and ps Nproc is defined at OS level to limit the number of processes […]<\/p>\n","protected":false},"author":27,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[229,59],"tags":[46,143,209,74],"type_dbi":[],"class_list":["post-3836","post","type-post","status-publish","format-standard","hentry","category-database-administration-monitoring","category-oracle","tag-linux-unix","tag-monitoring","tag-oracle-12c","tag-oracle-linux"],"acf":[],"yoast_head":"\nLinux: how to monitor the nproc limit - dbi Blog<\/title>\n<meta name=\"description\" content=\"You probably know about 'nproc' limits in Linux which are set in \/etc\/limits.conf and checked with 'ulimit -u'. But do you know how to handle the monitoring and be alerted when you're close the fixed limit?\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Linux: how to monitor the nproc limit\" \/>\n<meta property=\"og:description\" content=\"You probably know about 'nproc' limits in Linux which are set in \/etc\/limits.conf and checked with 'ulimit -u'. But do you know how to handle the monitoring and be alerted when you're close the fixed limit?\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2014-06-10T03:14:00+00:00\" \/>\n<meta name=\"author\" content=\"Oracle Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Oracle Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/\"},\"author\":{\"name\":\"Oracle Team\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee\"},\"headline\":\"Linux: how to monitor the nproc limit\",\"datePublished\":\"2014-06-10T03:14:00+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/\"},\"wordCount\":739,\"commentCount\":0,\"keywords\":[\"Linux\/UNIX\",\"Monitoring\",\"Oracle 12c\",\"Oracle Linux\"],\"articleSection\":[\"Database Administration & Monitoring\",\"Oracle\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/\",\"name\":\"Linux: how to monitor the nproc limit - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"datePublished\":\"2014-06-10T03:14:00+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee\"},\"description\":\"You probably know about 'nproc' limits in Linux which are set in \/etc\/limits.conf and checked with 'ulimit -u'. But do you know how to handle the monitoring and be alerted when you're close the fixed limit?\",\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Linux: how to monitor the nproc limit\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee\",\"name\":\"Oracle Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g\",\"caption\":\"Oracle Team\"},\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/oracle-team\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Linux: how to monitor the nproc limit - dbi Blog","description":"You probably know about 'nproc' limits in Linux which are set in \/etc\/limits.conf and checked with 'ulimit -u'. But do you know how to handle the monitoring and be alerted when you're close the fixed limit?","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/","og_locale":"en_US","og_type":"article","og_title":"Linux: how to monitor the nproc limit","og_description":"You probably know about 'nproc' limits in Linux which are set in \/etc\/limits.conf and checked with 'ulimit -u'. But do you know how to handle the monitoring and be alerted when you're close the fixed limit?","og_url":"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/","og_site_name":"dbi Blog","article_published_time":"2014-06-10T03:14:00+00:00","author":"Oracle Team","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Oracle Team","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/"},"author":{"name":"Oracle Team","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee"},"headline":"Linux: how to monitor the nproc limit","datePublished":"2014-06-10T03:14:00+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/"},"wordCount":739,"commentCount":0,"keywords":["Linux\/UNIX","Monitoring","Oracle 12c","Oracle Linux"],"articleSection":["Database Administration & Monitoring","Oracle"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/","url":"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/","name":"Linux: how to monitor the nproc limit - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2014-06-10T03:14:00+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee"},"description":"You probably know about 'nproc' limits in Linux which are set in \/etc\/limits.conf and checked with 'ulimit -u'. But do you know how to handle the monitoring and be alerted when you're close the fixed limit?","breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/linux-how-to-monitor-the-nproc-limit-1\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Linux: how to monitor the nproc limit"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/66ab87129f2d357f09971bc7936a77ee","name":"Oracle Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/f711f7cd2c9b09bf2627133755b569fb5be0694810cfd33033bdd095fedba86d?s=96&d=mm&r=g","caption":"Oracle Team"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/oracle-team\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/3836","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/27"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=3836"}],"version-history":[{"count":0,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/3836\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=3836"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=3836"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=3836"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=3836"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}