Home » Cloud
Category Archives: Cloud
Python Code to get EBS stats for SAP Systems
In this article, we have developed a Python script to simplify the process of retrieving AWS CloudWatch metrics for Elastic Block Store (EBS) volumes. The script takes input from a CSV file, containing metrics such as metric-name, VolumeId, START_TIME, and END_TIME, and uses the boto3 library to interact with AWS services.
By using this script, users can avoid manually executing individual AWS CLI commands for each metric and volume, making the process more efficient and less error-prone. The script iterates through the CSV file, calls AWS CloudWatch using boto3, and collects the required metric statistics, such as the Average value, for each metric and volume within the specified time range.
The output is then written back to a CSV file with the columns metric-name, VolumeId, Timestamp, and Average. This organized output allows users to easily analyze and further process the data for their specific use cases.
Users can customize the input CSV file with desired metrics and volumes, making it adaptable to various AWS environments and monitoring requirements.
SAMPLE - input.csv
metric-name,VolumeId,START_TIME,END_TIME
VolumeReadOps,vol-12345,2023-07-01T00:00:00,2023-07-02T00:00:00
VolumeWriteOps,vol-67890,2023-07-01T00:00:00,2023-07-02T00:00:00
BurstBalance,vol-54321,2023-07-01T00:00:00,2023-07-02T00:00:00
VolumeBytesRead,vol-98765,2023-07-01T00:00:00,2023-07-02T00:00:00
VolumeBytesWrite,vol-24680,2023-07-01T00:00:00,2023-07-02T00:00:00
CODE - sap_get_metric_statistics.py
import csv
import boto3
# Function to get CloudWatch metric statistics
def get_metric_statistics(metric_name, volume_id, start_time, end_time):
cloudwatch = boto3.client('cloudwatch')
response = cloudwatch.get_metric_statistics(
Namespace='AWS/EBS',
MetricName=metric_name,
Dimensions=[
{
'Name': 'VolumeId',
'Value': volume_id
},
],
StartTime=start_time,
EndTime=end_time,
Period=300,
Statistics=['Average']
)
return response['Datapoints']
# Main function
def main():
input_file = 'input.csv'
output_file = 'output.csv'
with open(input_file, 'r') as csvfile:
csvreader = csv.DictReader(csvfile)
next(csvreader) # Skip the header row
data = list(csvreader)
with open(output_file, 'w', newline='') as file:
csvwriter = csv.writer(file)
csvwriter.writerow(['metric-name', 'VolumeId', 'Timestamp', 'Average'])
for entry in data:
metric_name = entry['metric-name']
volume_id = entry['VolumeId']
start_time = entry['START_TIME']
end_time = entry['END_TIME']
datapoints = get_metric_statistics(metric_name, volume_id, start_time, end_time)
for datapoint in datapoints:
csvwriter.writerow([metric_name, volume_id, datapoint['Timestamp'], datapoint['Average']])
if __name__ == "__main__":
main()
SAMPLE - output.csv
metric-name,VolumeId,Timestamp,Average
VolumeReadOps,volume-1,2023-07-20 10:00:00,120.0
VolumeReadOps,volume-1,2023-07-20 10:05:00,130.0
VolumeReadOps,volume-1,2023-07-20 10:10:00,115.0
VolumeWriteOps,volume-1,2023-07-20 10:00:00,50.0
VolumeWriteOps,volume-1,2023-07-20 10:05:00,60.0
VolumeWriteOps,volume-1,2023-07-20 10:10:00,55.0
BurstBalance,volume-1,2023-07-20 10:00:00,75.0
BurstBalance,volume-1,2023-07-20 10:05:00,80.0
BurstBalance,volume-1,2023-07-20 10:10:00,70.0
VolumeBytesRead,volume-1,2023-07-20 10:00:00,2000.0
VolumeBytesRead,volume-1,2023-07-20 10:05:00,2200.0
VolumeBytesRead,volume-1,2023-07-20 10:10:00,1900.0
VolumeBytesWrite,volume-1,2023-07-20 10:00:00,1500.0
VolumeBytesWrite,volume-1,2023-07-20 10:05:00,1700.0
VolumeBytesWrite,volume-1,2023-07-20 10:10:00,1400.0
Python3 AWS CLI script to create machines
Python3 script to create AWS VM using cli, the code reads the input from a file called “input.txt” with a different format, where each line contains the name of the value and the actual value separated by a tab:
import boto3
# Set your AWS profile and region
AWS_PROFILE = "your-profile"
AWS_REGION = "your-region"
# Read input from file
with open("input.txt", "r") as file:
lines = file.readlines()
# Extract instance details from input
INSTANCE_TYPE = ""
IMAGE_ID = ""
SUBNET_ID = ""
SECURITY_GROUP_ID = ""
KEY_NAME = ""
TAG_NAME = ""
TAG_ENVIRONMENT = ""
TAG_OWNER = ""
# Parse input lines
for line in lines:
name, value = line.strip().split("\t")
if name == "INSTANCE_TYPE":
INSTANCE_TYPE = value
elif name == "IMAGE_ID":
IMAGE_ID = value
elif name == "SUBNET_ID":
SUBNET_ID = value
elif name == "SECURITY_GROUP_ID":
SECURITY_GROUP_ID = value
elif name == "KEY_NAME":
KEY_NAME = value
elif name == "TAG_NAME":
TAG_NAME = value
elif name == "TAG_ENVIRONMENT":
TAG_ENVIRONMENT = value
elif name == "TAG_OWNER":
TAG_OWNER = value
# Create an EC2 resource using the specified profile and region
session = boto3.Session(profile_name=AWS_PROFILE, region_name=AWS_REGION)
ec2_resource = session.resource('ec2')
# Launch the EC2 instance
instance = ec2_resource.create_instances(
ImageId=IMAGE_ID,
InstanceType=INSTANCE_TYPE,
SubnetId=SUBNET_ID,
SecurityGroupIds=[SECURITY_GROUP_ID],
KeyName=KEY_NAME,
TagSpecifications=[
{
'ResourceType': 'instance',
'Tags': [
{'Key': 'Name', 'Value': TAG_NAME},
{'Key': 'Environment', 'Value': TAG_ENVIRONMENT},
{'Key': 'Owner', 'Value': TAG_OWNER}
]
}
]
)[0]
# Wait for the instance to be in the "running" state
instance.wait_until_running()
print(f"Instance {instance.id} launched successfully!")
the “input.txt” file, contains the name of the value and the actual value separated by a tab (\t), each line is parsed and the actual value to the corresponding variable based on the name is assigned.
an example of “input.txt” file structure is as below:
INSTANCE_TYPE t2.micro
IMAGE_ID ami-12345678
SUBNET_ID subnet-12345678
SECURITY_GROUP_ID sg-12345678
KEY_NAME your-key-pair-name
TAG_NAME MyInstance
TAG_ENVIRONMENT Production
TAG_OWNER John Doe
aws cli – snapshots listing and correlations
Today i was working on one curious issue, and I created few AWS CLI Snippets. I think the best use case of below snippets (in sequence) will be when we deregister an AMI, as this will result in detachment of snapshots attached to it and the snapshots can be deleted, but there can be cases when we want to preserve the snapshots, usually when we have huge amount of snapshots then listing and searching for snapshot-ids in AWS console may gets timed out, in those cases you can use below snippets to get the desired results.
1. run below to get the list of volume-ids attached to an instance, in this we wanted to know VolumeId, InstanceId, DeviceName, and any relevant tag like Name tag, etc.
aws ec2 describe-instances --filters "Name=tag:Name,Values=<hostname>" --query "Reservations[].Instances[].{deviceID:BlockDeviceMappings[].Ebs.VolumeId,deviceName:BlockDeviceMappings[].DeviceName,InstanceId:InstanceId,Name:Tags[?Key=='Name']|[0].Value}"
2. copy one Volume-Id from above step, and add in below section, in this step we will be finding the date and time of the snapshots, we will be copying the date and time of the restore snapshot-ids to be restored, note if you are restoring an instance then it is advisable to restore all attached volumes, but just to bring the instance up you can restore root volume (/dev/sda1) also.
aws ec2 describe-snapshots --filters "Name=volume-id,Values=<volume-id-from-step-1>" --query "Snapshots[*].{DateTime:StartTime,Volume:VolumeId,SnapshotId:SnapshotId,Description:Description}" --output table
3. now we will use the date and time from above and will run to get the exact snapshot ids on that time for that volume-id.
aws ec2 describe-snapshots --filters "Name=volume-id,Values=<volume-id-from-step-1>" "Name=start-time,Values=2021-08-11T10:19*<this-is-date-from-step-2>" --query "Snapshots[].{DateTime:StartTime,Volume:VolumeId,SnapshotId:SnapshotId,Description:Description}" --output table
4. now run the step-3 for all volume-ids we got from step-1, from step-1 you will also see the association that is what volume-ids were attached as what block device like /dev/sda1, etc.