Automated Server Monitoring and Alerting System on EC2

Here’s a detailed step-by-step guide to create a system that monitors server health (CPU, memory, disk usage) on EC2 instances and automatically sends alerts via Amazon SNS when thresholds are exceeded. but i creted only CPU UTILIZATION in this project but i ma provide (memory, disk usage)

Step 1: Set Up Your EC2 Instance

1.1 Launch an EC2 Instance

1. Log in to the AWS Management Console:

Go to AWS Console.

Sign in with your AWS account credentials.

2. Navigate to EC2 Dashboard:

In the AWS Management Console, search for "EC2" and click on EC2 under the Services section.

3. Launch a New EC2 Instance:

On the EC2 Dashboard, click on the Launch Instance button.

Give your instance a Name (e.g., “ServerMonitoringInstance”).

4. Choose an Amazon Machine Image (AMI):

Select Amazon Linux 2023 or Amazon Linux 2 (recommended for compatibility with most monitoring tools).

5. Choose an Instance Type:

Select an instance type based on your requirements.

For a basic monitoring system, t2.micro (which is eligible for the free tier) is sufficient.

6. Configure Key Pair (Login):

In the Key pair (login) section, either create a new key pair or choose an existing one.

If creating a new key pair:

Choose Create new key pair.

Select RSA or ED25519 as the key type.

Click on Create key pair.

The private key (.pem file) will be downloaded to your computer.

7. Network Settings (Security Group Configuration):

Set up your security group to allow SSH access and optionally HTTP/HTTPS if needed for web access.

Create a new security group or choose an existing one.

Ensure that SSH (port 22) is open to allow you to connect.

You can restrict the source IP to only your IP address for better security.

8. Example Security Group rules: SSH: Port 22, Source: Your IP (for secure access).

HTTP: Port 80, Source: Anywhere (if you want web access).

HTTPS: Port 443, Source: Anywhere (if you use SSL).

9. Configure Storage:

For a simple monitoring setup, the default 8 GB of storage is enough.

If you anticipate more intensive monitoring tasks or logging, you can increase the size.

Launch the Instance:

Review all settings and click Launch Instance. Wait for the instance to be launched. This may take a few minutes.

cheak the status, instance ID , public IP adress

connect to instance

Step 2: Install Required Packages

Update the EC2 Instance To ensure that your instance has the latest security patches and updates, it’s recommended to update the system before installing any software.

sudo yum update -y

Install the CloudWatch Agent The Amazon CloudWatch Agent is needed to collect CPU, memory, and disk usage metrics from your EC2 instance and send them to Amazon CloudWatch for monitoring.

sudo yum install amazon-cloudwatch-agent -y

This command installs the CloudWatch agent and its dependencies. The CloudWatch agent will be used to collect and send system metrics (e.g., CPU, memory, and disk usage) to CloudWatch.

Verify the Installation:

After the installation is complete, you can verify that the CloudWatch agent is installed by checking its version:

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -version

Install Shell Scripting Tools (Optional) If you plan to create custom shell scripts for additional monitoring or automation, you might want to install some shell scripting utilities. While Amazon Linux comes with many tools pre-installed, you can install additional packages as needed.

sudo yum install vim -y
sudo yum install nano -y
sudo yum install htop -y
sudo yum install bc -y

Confirm Installation of Optional Tools:

Check the version of any installed tools to confirm they are ready for u

vim --version
nano --version
htop --version
bc --version

Step 3: Configure the CloudWatch Agent

Create a CloudWatch Agent Configuration File AWS provides a built-in configuration wizard to help you generate a config.json file for the CloudWatch agent. This file defines what metrics the agent will collect and send to CloudWatch.

Run the CloudWatch Agent Configuration Wizard:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

Follow the Prompts:

The wizard will ask you a series of questions to determine which metrics to monitor and where to send the data. Answer the

questions based on your needs:

Operating system: Choose Linux.

Metrics to collect: You can choose to collect CPU, memory, and disk usage metrics.

CloudWatch Log Group: Select a log group name or create a new one.

Destination: Choose EC2 for monitoring EC2 instances.

you follow the read underline meand (default choose)

Locate the Configuration File:

/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

you copy this command and paste lof file path and also paste you EC2 instnace id

you only not mention X-reay Demon configuration you only press ENTER button

you not wirte [AmazonCloudWathc-linux] you press ENTER button,

you not write [ us-east-1] you press Ensure button,

you not menstion access key and access secret key , you only press button you move on space.

Start the CloudWatch Agent Once the configuration file is ready, you can start the CloudWatch agent to begin collecting and sending data to CloudWatch.

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
    -a start -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s

This command tells the agent to use the configuration file you created and start collecting metrics on your EC2 instance.

Verify the CloudWatch Agent is Running:

To check if the CloudWatch agent is running, use the following command:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status

The output should show the agent's status as (running) if everything is working correctly.

Step 4: Set Up CloudWatch Alarms

Navigate to CloudWatch Alarms:

1. Go to the AWS Management Console.

In the search bar, type "CloudWatch" and click on CloudWatch to open the CloudWatch dashboard.

In the left-hand navigation pane, click on Alarms.

Click Create Alarm to start the process.

2. Select the Metric:

Click Select metric and follow the prompts to find the metrics you want to monitor.

\\For CPU Utilization:

In the Browse tab, select EC2 > Per-Instance Metrics > CPU Utilization.

Choose your EC2 instance from the list, then click Select metric. Set the threshold value.

For example, set the alarm to trigger if CPU Utilization exceeds 80% for 5 minutes.

\\For Memory Usage:

Since memory usage is not natively available from EC2, use the custom metrics sent by the CloudWatch Agent.

Go to Browse > CWAgent > InstanceId > Memory > Usage.

Choose your instance and select the Memory Utilization metric.

Set a threshold (e.g., alert if memory usage exceeds 85%).

\\For Disk Usage:

Select the metrics for disk space sent by the CloudWatch agent.

Navigate to CWAgent > InstanceId > Disk > UsedPercent.

Set a threshold (e.g., alert if disk space usage exceeds 80%) . 3. Configure Alarm Settings:

For each metric (CPU, Memory, Disk), set the following options:

Period: Set the period to 5 minutes (or any preferred interval).

Threshold type: Set the threshold for when the alarm should trigger (e.g., 80% CPU utilization).

you write search on metric ( EC2, CPU, Utilization and copy instance id and paste)

you cheak your CPU utilization becuase my utilization graph percentage is 11 then i am set 12 value

you set your value depend ghraph percentage and near set value

1st create SNS topic

Step 5: Configure Amazon SNS for Alerts

Amazon SNS (Simple Notification Service) allows you to send notifications to email addresses, phone numbers, or other endpoints when the alarm is triggered.

1. Navigate to SNS in the AWS Console:

In the AWS Management Console, type “SNS” in the search bar and click on Simple Notification Service (SNS).

2. Create a New SNS Topic:

Click on Topics from the left-hand menu, then click Create Topic.

Topic Type: Choose Standard (recommended for most use cases).

Name your topic (e.g., “EC2-Server-Health-Alerts”).

Click Create Topic.

3. Subscribe to the Topic:

After creating the SNS topic, you need to subscribe an endpoint (like your email or phone number) to receive notifications.

Click on Create subscription.

Topic ARN: Select the SNS topic you just created.

Protocol: Choose how you'd like to receive notifications.

You can choose Email for simple notifications or SMS for text alerts.

Endpoint: Enter the email address or phone number where you want to receive alerts.

Click Create subscription.

4. Confirm the Subscription:

If you selected Email as the protocol, you’ll receive a confirmation email. Click the confirmation link to activate the subscription.

set SNS topic in send a notification

create alarm

click the alarm and watch monitor

Step 6: Automate Health Checks with Shell Scripting

Create a Monitoring Script

Create the Shell Script:

SSH into your EC2 instance if you're not already connected.

Create a new shell script file (e.g., monitor_health.sh)

sudo nano /monitor_health.sh

COPY SNS Topic ARN AND paste

CPOY DISTK USAGE : dvetmpfs

paste SNS ARn & Disk usage

Add the Following Code to the Script: This script will check memory and disk usage and send alerts via SNS if the thresholds are exceeded.

#!/bin/bash

# Check memory usage (in percentage)
MEMORY_USAGE=$(free -m | awk 'NR==2{printf "%.2f", $3*100/$2 }')

# Check disk usage (in percentage, replace /dev/nvme0n1p1 with the correct disk partition)
DISK_USAGE=$(df -h | grep '/dev/nvme0n1p1' | awk '{print $5}' | sed 's/%//')

# Define thresholds
MEMORY_THRESHOLD=80
DISK_THRESHOLD=90

# Send alerts if memory usage exceeds the threshold
if (( $(echo "$MEMORY_USAGE > $MEMORY_THRESHOLD" | bc -l) )); then
    aws sns publish --topic-arn arn:aws:sns:region:account-id:ServerHealthAlerts \
    --message "Memory usage is above threshold: $MEMORY_USAGE%"
fi

# Send alerts if disk usage exceeds the threshold
if [ "$DISK_USAGE" -gt "$DISK_THRESHOLD" ]; then
    aws sns publish --topic-arn arn:aws:sns:region:account-id:ServerHealthAlerts \
    --message "Disk usage is above threshold: $DISK_USAGE%"
fi

Modify the Script:

Replace /dev/nvme0n1p1 with the correct disk partition on your instance. You can use the df -h command to check your partition.

Replace arn:aws:sns:region:account-id:ServerHealthAlerts with your actual SNS Topic ARN (from Step 5).

Make the Script Executable:

Save and exit the file.

Make the script executable by running the following command

sudo chmod +x /path_to_your_script/monitor_health.sh

Set Up a Cron Job for Regular Monitoring

Install the cronie Package

sudo yum update -y
sudo yum install cronie -y
sudo systemctl start crond
sudo systemctl enable crond
sudo systemctl status crond

Edit the Crontab

crontab -e

Add Your Cron Job: For example, if you want to run a script every 5 minutes, you would add this line to the crontab:

*/5 * * * * /monitor_health.sh

Verify the Cron Job: To ensure that the cron job has been saved, you can list all your active cron jobs with:

crontab -l

7 Trigger an Alarm

Simulate High CPU Usage To create high CPU usage, you can use a tool like stress or run a CPU-intensive process.

Install the stress tool:

If stress is not installed, you can install it with

sudo yum install stress -y

Run stress to Simulate High CPU Usage:

Use the following command to simulate high CPU usage for 5 minutes:

stress --cpu 2 --timeout 300

your project is succesfully run but you wait 3min to 4 min beause result you see after 3min

configuration you have notification

you delete / terminated all service becuse work is completed