Configure a Policy-action (new name for watch-notification) to send a JMX notification when the monitored WebLogic Server is having stuck threads or high memory usage. We will use a simple JMX notification listener program which can process WLDF JMX notifications. In this case, it will simply print the contents of the notification.
For simplicity, we will have only two servers in the domain, AdminServer (admin-server) and Server1 (managed-server). We will configure WLDF on the Server1 to send a JMX notification when the Heap Free of the managed-server Server1 (actually, any server it is targeted to within the domain) is less than 20% or as soon as there is a Stuck thread.
We will use WebLogic Server Administration Console to configure WLDF. It can also be done with WLST. Console provides a nice feature to Create Repeatable Configuration Scripts, which can be handy for building such scripts. Configuration steps for this case consist of:
1. Create a Diagnostic Module.
2. Enable periodic metrics collection in the Diagnostic module.
3. Create a JMX notification type.
4. Create a policy to detect if the WebLogic Server is having stuck threads
5. Create a policy to detect high memory usage on the WebLogic Server
6. Target the Diagnostic module to the WebLogic Server server1
We will go over these steps and see how to configure WLDF using Console for this task.

1. Create a Diagnostic Module.

a) Log into Console and acquire Edit Lock so we can add new configuration to WebLogic Server.
b) From the left navigation tree, open the Diagnostics node and click on Diagnostic Modules. Console will show a table of existing Diagnostic modules within the domain. Click the New button to create a new diagnostic module. Call it myWLDF. Click OK to create the module. At this point, we have an empty myWLDF diagnostic module.

2. Enable periodic metrics collection in the Diagnostic module.

a) Click on the myWLDF module link in the table of Diagnostics modules.
b) Click on Collected Metrics sub-tab under Configuration tab.
c) Check the Enabled checkbox and set the Sampling Period to 10000 (10 seconds). Click Save.

3. Create a JMX notification type.

a) Configuring a policy/actions (Watch and Notifications in earlier versions) has two aspects. The first aspect is a policy (watch rule) which specifies the condition that WLDF will check. The second aspect is the set of actions (notifications) that will be sent when the rule condition is met. Console provides configuration assistants to make the configuration task easier. To create an action type:
b) Click Policies and Actions sub-tab under Configuration tab.
On the Actions sub-tab, click New in the Actions table.
c) Select “JMX Notification” for the notification type from the drop down list and click Next.
d) Give a name to the notification type (myJMX)
e) Keep the Notification Type to its default value
f) Check the Enable Notification checkbox and click OK to create the notification type.

4. Create a policy to detect servers having stuck threads

Now, we will create the policy rule based on runtime mbean data. Specifically, we will use the StuckThreadCount attribute on the WorkManagerRuntime mbeans. For each server within the domain, there is a WorkManagerRuntime mbean in the domain runtime mbean server. The StuckThreadCount attribute reflects the current number of stuck thread on the server. We will configure a rule which will fire as soon one server in the targets list is having a stuck thread.
a) Click on the Policies sub-tab -> Configuration. Click New in the Policies table.
b) Set Policy Name to WatchServerStuckThread. Select Collected Metrics for Policy Type, check Enabled checkbox and click Next.
c) Set the Policy Expression as below and Click Next:

wls.runtime.query('com.bea:Type=WorkManagerRuntime,*','StuckThreadCount').stream().anyMatch( x -> x > 0 )

Note that the syntax of Policy Expressions has changed from the previous WebLogic Versions. Starting from this version, the policy expression is to be provided in Java Expression Language (EL).

d) Keep The Frequency to “Every N Seconds” and Click Next
e) Set the Repeat parameter to 5 and click Next
f) In this wizard keep the default and click Next
g) In the diagnostic Actions part, in the available actions, select the myJMX created earlier and move it to the “chosen” actions.
h) Click Finish

5. Create a policy to detect high memory usage on the WebLogic Server

Now, we will create the policy rule based on runtime mbean data. Specifically, we will use the heapFreePercent attribute on the JVMRuntime mbeans. For each server within the domain, there is a JVMRuntime mbean in the domain runtime mbean server. The heapFreePercent attribute reflects the current free heap size on the server. We will configure a rule which will fire as soon one server in the targets list is having less than 20% heap size free.
a) Click on the Policies sub-tab -> Configuration. Click New in the Policies table.
b) Set Policy Name to WatchServerHeapFree. Select Collected Metrics for Policy Type, check Enabled checkbox and click Next.
c) Set the Policy Expression as below and Click Next:

wls.runtime.serverRuntime.JVMRuntime.heapFreePercent < 20

d) Keep The Frequency to “Every N Seconds” and Click Next
e) Set the Repeat parameter to 5 and click Next
f) In this wizard keep the default and click Next
g) In the diagnostic Actions part, in the available actions, select the myJMX created earlier and move it to the “chosen” actions.
h) Click Finish
6. Target the Diagnostic module to the WebLogic Server server1
a) Click on the Targets TAB
b) In the list of possible Targets, select the Server1 and click on Save
c) Activate Changes

Receiving Notifications

WLDF sends a JMX notification on a specific WLDF runtime mbean, whose ObjectName is of the form:
com.bea:Name=DiagnosticsJMXNotificationSource,ServerRuntime=$SERVER,Type=WLDFWatchJMXNotificationRuntime,WLDFRuntime=WLDFRuntime,WLDFWatchNotificationRuntime=WatchNotification
where $SERVER is the name of the WebLogic Server instance. For our case (Server1), it is:
com.bea:Name=DiagnosticsJMXNotificationSource,ServerRuntime=Server1,Type=WLDFWatchJMXNotificationRuntime,WLDFRuntime=WLDFRuntime,WLDFWatchNotificationRuntime=WatchNotification
By registering for JMX notifications on this mbean, a client program can listen to generated notifications.
We will use the JMXWatchNotificationListener.java provided in the Oracle WLDF documentation (see references). It is a simple notification listener for WLDF JMX notifications. It simply prints the contents of received notification, but can be easily adapted to perform other actions.

A sample Java code of such listener can be downloaded from the Oracle WebLogic Diagnostic Framework documentation (here)

To run it for this blog sample run:

java JMXWatchNotificationListener vm01 7006 weblogic Welcome1 Server1

Note: The WebLogic Managed Server named Server1 is listening on port 7006.

Sample WLDF message sent when the WebLogic Server is having Stuck Threads

Notification name: myJMXNotif called. Count= 79.
Watch severity: Notice
Watch time: Apr 24, 2018 12:08:35 PM CEST
Watch ServerName: Server1
Watch RuleType: Harvester
Watch Rule: wls.runtime.query('com.bea:Type=WorkManagerRuntime,*','StuckThreadCount').stream().anyMatch( x -> x > 0 )
Watch Name: WatchServerStuckThread
Watch DomainName: wldf_domain
Watch AlarmType: None
Watch AlarmResetPeriod: 60000

Sample WLDF message sent when the WebLogic Server heap free becomes low (less than 20%)

Notification name: myJMXNotif called. Count= 114.
Watch severity: Notice
Watch time: Apr 24, 2018 12:11:45 PM CEST
Watch ServerName: Server1
Watch RuleType: Harvester
Watch Rule: wls.runtime.serverRuntime.JVMRuntime.heapFreePercent < 20
Watch Name: WatchServerLowHeapFreePercent
Watch DomainName: wldf_domain
Watch AlarmType: None
Watch AlarmResetPeriod: 60000