Linux CounterMeasures

As a quick reminder, CounterMeasures are automated responses to specific events that the Panopta agent detects on your infrastructure. We provide a handful of automated actions out-of-the-box and also allow you to write your own. Regardless of the route you choose, you will be leveraging the Panopta agent as the conduit for the process.

Enabling CounterMeasures

Agent Requirements

CounterMeasures are available beginning in Linux agent version 2018.14. If you're using an agent version older than this, check out this article for more information upgrading your agent.

Fresh Install

If you're performing a new agent installation, you can enable CounterMeasures simply by adding --countermeasures onto the agent 1-line installer

curl -s https://packages.panopta.com/install/linux_agent_install.py | python3 /dev/stdin --customer-key YOUR-CUSTOMER-KEY --countermeasures 

The agent will now be installed and CounterMeasures will be ready to go on your instance. You can confirm that CounterMeasures have successfully been enabled by visiting the specific instance in the Panopta Control Panel, and locating the CounterMeasures Enabled datapoint in the Agent Status information block on the right side of the page.

Existing Agent Install

Open your agent config file, located at /etc/panopta-agent/panopta_agent.cfg. Add the enable_countermeasures flag as a new line to the file.

enable_countermeasures=true

Next, run python /usr/bin/panopta-agent/panopta_agent.py --rebuild-metadata. This command resends metadata about your agent installation up to Panopta. You can also do this from the control panel.

The next time the agent runs (every 60 seconds), it will pick up the new flag and will be ready to go.

Configuring a CounterMeasure

CounterMeasures are tied to metric thresholds, which means they're configured together as well. Let's walk through how to add a CounterMeasures to capture top when CPU usage elevates.

  1. In the Panopta control panel, navigate to an instance with CounterMeasures enabled.
  2. Scroll down to the monitoring configuration table and locate the Linux: CPU section
  3. Edit the CPU: Usage percentage - Total metric
  4. In the slide out pane, expand the Thresholds & CounterMeasures section
  5. If you don't already have a threshold set, go ahead and set one.
  6. At the bottom of your configured threshold, click + Add CounterMeasure
  1. Configure your two CounterMeasure options
    • Delay: this is how long to wait after the threshold is triggered before running the CounterMeasure. For now, select 0 minutes, which means it will fire immediately.
    • Action: this is the specific CounterMeasure that will be run. Select top, which is provided out of the box.
    • Approval: require approval before the CounterMeasure is run. You can read more about approval below.
    • Send to Slack: if you've previously sent an alert to Slack related to this incident, we'll also send any CounterMeasure output to this #channel.
  2. Hit save.

Now you're good to go. When the threshold is crossed, the countermeasure will be triggered. We'll run top, collect the output, and display it in the log for the generated incident.

CounterMeasures and external checks

External monitoring checks (HTTP, HTTPs, Ping, etc) can also be used to trigger CounterMeasures. The agent still must be running on the instance, but when the external check threshold is crossed, it will trigger a CounterMeasure (e.g., restart Apache) on the instance.

Viewing CounterMeasure output

The output (if any) from your CounterMeasure action is available from a number of locations.

CounterMeasure Details Panel

Throughout the ControlPanel, you can open the CounterMeasures detail panel by clicking the CounterMeasures indicator icon, which is the beaker.

The beaker icon can be found in a number of locations - virtually anywhere an incident may be listed.

  • Dashboards (Active Incident and Incident History widgets)
  • Incident History Page
  • Recent Incidents Tab on the Instance Details page
  • Active Incidents Modal

Clicking the icon will open the CounterMeasures detail panel. It containers, among either things, any output the CounterMeasure returned after running.

Incident Details Page

CounterMeasures output is also available from the Incident details page. In the top navigation, select Incidents. From the table, select the incident you're interested in (you'll also noticed the beaker icon in the row). From the Incident details page, select Outage Log. In the log, you will find details about which CounterMeasures have been executed and what output the returned.

CounterMeasure States

CounterMeasures have a simple, linear lifecycle. Each lifecycle state is covered below

  • Pending: state of an action that is scheduled to execute in the future and does not require approval or has already been approved. The beaker icon will appear grey.
  • Pending Approval: state of an action that is scheduled to execute in the future but will not do so until approved. The beaker icon will glow orange.
  • In Process: state of an action that has been synced down to the agent to execute. The beaker icon will appear grey.
  • Complete: state of an action that has executed successfully. The beaker icon will appear green.
  • Error: state of an action that attempted to execute but returned an error while executing. The beaker icon will appear red.
  • Skipped: state of an action that was at one time pending but whose underlying incident was resolved before executing. The beaker icon will appear grey.

Approval

Certain situations may arise where you don't want a CounterMeasure to execute unless someone gives the final go-ahead. This is supported via the Approval option.

When configuring your CounterMeasure, check the Require Approval option. Anywhere the incident is displayed and the beaker indicator is present, the beaker will glow orange, indicating approval is required for at least one CounterMeasure on the incident. Clicking it will show a modal where you can approve the appropriate CounterMeasures. CounterMeasures that require approval will stay queued until approved; they will not block other CounterMeasure and will not run until their configured timing (e.g., you can approve them early and they'll still run at the proper time).

Using CounterMeasures in Templates

CounterMeasures work in templates the same way they would on a regular instance, except for one difference - in templates, the list of available actions is a union of all the CounterMeasures used across your account. On instances, the only actions available are the ones that have been reported by the instance.

If you attempt to apply a template with a CounterMeasure to an instance that has not reported that CounterMeasure as present, it will not be added to the monitoring configuration for that metric.

Utilizing Remote Actions

Centrally managing your custom CounterMeasure plugins is the best way to ensure consistency across your infrastructure monitoring. To help with this, you can add the countermeasures_remote_plugins parameter to your agent manifest file that points to an archive of agent plugins. When the agent is installed, it will download and extract the directory in the proper location. Zip, tar, and python file types are supported.

If you would like updates to your CounterMeasures plugins to automatically be applied, you can optionally set countermeasures_refresh_plugins to the number of hours for the refresh interval. This will help with keeping all your servers up-to-date.

customer_key = XXX-XXX-XXX-XXX
enable_countermeasures=true
countermeasures_remote_plugins = https://s3.amazonaws.com/some-s3-bucket/custom-plugins.zip
countermeasures_refresh_plugins = 6

Post agent installation

If you'd like to download a set of remote plugins after the installation process has been completed, you can run the following command

python /usr/bin/panopta-agent/countermeasure.py install_plugins --url URL

Then, rebuild your agent metadata

python /usr/bin/panopta-agent/panopta_agent.py --rebuild-metadata

Once the metadata rebuild is complete, your plugins will be available for use in the Panopta control panel.