How to set up a remediation action due to a thread pool size issue.
Set up a policy within the node that states if any of the request counts get above 800 per minute we can consider it to be critical.
Configuration to handle this issue.
1. Go to the Health Rule page under ''Alert & Respond'' in the Application Dashboard
2. Create a new rule that has ''Node Health - JMX (connection pools, thread pool, etc)'' selected
3. Select the JMX object to be monitored from the ''Affects'' section in the Create Health Rule window.
4. Select the Tier
5. You do have the option to apply this to all nodes in the application.
To set up the condition:
1. Critical Condition
2. Click ''Add Condition''
3. Name condition
4. Click ''Select metric''
5. Select request count
6. Save
7. Switch to ''Specific Value'' in ''is'' dropdown.
8. In the next field set up the condition at the number of requests desired
9. Click Save
Warning Condition:
Set up in the exact same manor except lower thread count
Next step is to set up the remediation action
1. Go to the ''Actions'' page under ''Alert & Respond'' on the Application Dashboard
2. Click ''Create Action''
3. At the pop up Create Action window select ''Run a script or executable on problematic Nodes'' under Remediation
4. Name the action
5. Input the path to the script
6. To get approval before action is taken click the ''Require approval before executing this..'' box
7. Then input the email address or SMS settings of the approver
Final task is to create a policy to trigger this action when this health rule is violated.
1. Go to ''Policies'' under the Alert & Respond section in the Application Dashboard
2. Click ''Create Policy''
3. In the ''Trigger'' section select the health rule previously set up for this issue
4. In the ''Actions'' tab select the action that was created for remediation.
5. Save
Whenever a remediation policy gets activated it will appear under Events in the application dashboard. Drill down from there to get additional details