Automation of a Security Agent Installation with Automation as Code

AWS DEV SEC OPS

Introduction

In this article, I’ll walk through provisioning EC2 instances using Terraform and automating the deployment of security agents on EC2 instances.

AWS Services and Technology

Note: If you follow the steps in this article and change any object or file names, please ensure you modify the commands given herein with the names you used.

Use Case

Your team has been asked to deploy EC2 instances in an automated way (Infrastructure as code). You also need to install a security agent in each EC2 instance and set up notifications in a manner that promotes EC2 provisioning standardization and enforces the organization’s security standards.

Solution Architecture

The best way to ensure that EC2 instances are created according to standard is to use the infrastructure-as-code design pattern with Terraform and AWS APIs. Maintaining security standards through automation also ensures all EC2 instances meet the defined security standards of the organization.

Solution Architecture

Implementation

Download the Terraform files here to provision the EC2 instances for this exercise. Unzip the files, open the file named “main.tf” and provide the VPC Id, Subnet Id, and SSH Key pair file name in the variable definitions with “change me” as the value.

Terraform code snippet

An IDE is not required to edit this file, but it can make editing Terraform files easier. If you don’t have Visual Studio Code installed, download it from https://code.visualstudio.com/download. Once installed, add the HashiCorp Terraform extension to see the syntax highlighting.

Open AWS Cloud Shell and upload your edited Terraform code to the AWS Cloud Shell. Install Terraform using the commands:

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo
sudo yum -y install terraform

Run the Terraform code using the following commands:

terraform init
terraform plan -out tf.plan
terraform show -no-color tf.plan > tfplan.txt
terraform apply tf.plan

Reviewing the Terraform plan output before you apply the configuration change is good practice to ensure all the requirements will be implemented as expected. Explore more about Terraform at https://learn.hashicorp.com/terraform.

If you haven’t already done so, an AWS IAM Role must be created to allow the AWS Systems Manager to use the AWS Simple Notification Service (SNS) service.

• Name: SystemsManagerToSNS
• Service: Systems Manager
• Description: Allows Systems Manager to call AWS SNS services on your behalf
• Add Policy: AmazonSNSFullAccess

Next, we create the SNS standard Notification Topic and create a subscription. For this exercise, I selected the email protocol. The other available protocols could send notifications to other monitoring systems, such as DataDog, Dynatrace, or Splunk, by writing files to an S3 bucket or triggering a Lambda to submit the output to a third-party API.

AWS SNS Topic and Subscription

Now to the fun stuff. Using the AWS System Manager Quick Setup, create a new configuration:

• Type: Host Management
• Region: Current Region
• Targets: Manual; select the EC2 instances created by the Terraform code

Note: Other options for selecting the Targets are All instances, Tag — the key-value pair for the tag associated with the EC2 instances you want to target, and Resource Group. These settings allow new EC2 instances to be dynamically added to the target list.

Creating the configuration with the settings above will:

• Enable Systems Manager Explorer in all targeted accounts and Regions.
• Deploy IAM roles that enable State Manager to invoke Automation documents that apply selected configuration options.
• Create a State Manager association for each configuration option you have selected.
• Attach instance profiles or IAM roles with required Systems Manager permissions to targeted instances.

A Common Issue

The EC2 instances don’t show up in the Managed Instances listing of the Systems Manager Console. So what happened?

To Troubleshoot this issue, log in to one of the EC2 instances and run the following commands:

sudo su –
cd /var/log/amazon/ssm/
ls -ltr
cat amazon-ssm-agent.log

The most command error message found in the log when the EC2 instances are not listed in the Managed Instances of the System Manager Console is EC2RoleRequestError: no EC2 instance role found.

Root Cause: The SSM agent will use an instance role to securely communicate with the EC2 Systems Manager. Usually, this is expected to be set up when the instance is initially launched. This exercise adds the Systems Manager process after the EC2 creation. The “AmazonSSMRoleForInstancesQuickSetup” IAM Role is added to the EC2 Instance during the Quick Setup process. Yet, the EC2 instances are not in sync.

Solution: Reboot each instance; the instance manager scripts will execute properly during reinitialization.

Automating the Security Agent Installation

For this exercise, I used a mock security agent. Other agents, such as AWS Inspector, CloudWatch, or a third-party security agent, could be substituted.

Using the AWS Systems Manager Run Command, I created a command document:

• Command Document Type: AWS-RunShellScript
• Command Parameters | Commands:

sudo wget -q https://raw.githubusercontent.com/slaceved2/Articles/main/Terraform_AWS_SSM_SNS/install_security_agent.sh -P /tmp --backups=1
sudo chmod +x /tmp/install_security_agent.sh
sudo /tmp/install_security_agent.sh
ls -ltr /usr/bin/security_agent

• Target Selection: Manual, select the EC2 instances created by the Terraform code.
• Output options: enables an S3 bucket and CloudWatch logs to receive the output.

Note: Other options for selecting the Targets are Tag — the key-value pair for the tag associated with the EC2 instances you want to target, and Resource Group. These settings allow new EC2 instances to be dynamically added to the target list.

To launch the command document, click the Run button. The command document is stored in the history so you can check for error messages, and it can be triggered again.

Run Command Results

Run Command Output

Email Notification

Summary

The example above walked through provisioning a group of EC2 instances using Terraform and setting up a mock security agent using AWS Systems Manager with an SNS email notification. Organizations can use automation and the infrastructure-as-code design pattern to maintain security standards and ensure ongoing supportability on the AWS platform. If you include AWS Resource Grouping or Tagging you can also use the AWS Systems Manager to automate tasks such as patching and upgrades.

If you enjoyed reading this article or found it helpful, don’t forget to click the like button or follow me. You can also reach out to me on LinkedIn.

References

Troubleshoot why an Amazon EC2 instance doesn’t appear under Managed Instances in the Systems Manager console | AWS re:Post (repost.aws)

How to Simplify Security Assessment Setup Using Amazon EC2 Systems Manager and Amazon Inspector | AWS Security Blog