Metrics Agent¶

Overview¶

Metrics agent is usually an executable script/command responsible to collect metrics from the required services running on the cloud and stream them to the respective metrics service created on CloudAEye.

This topic provides complete details about the the metrics agent script deployed on your cloud environment to collect metrics from the selected services.

Prerequisites¶

CloudAEye makes use of python to automate the process of deploying an agent on to your cloud environment.

User must have python3 installed on the system

How it Works?¶

AWS ECS(Fargate)¶

CloudAEye's ECS metrics agent script does the following:

Fetches the subnet and security group details of the VPC
Regsiters a new task definition for the agent
Runs the agent as a standalone task on the ECS cluster

How boto3 is used¶

To perform the above steps automatically, the agent script makes use of AWS boto3 python SDK. The AWS services accessed by the SDK are:

AWS EC2
- Describe Subnets Fetches all the subnets under the given VPC
- Describe SecurityGroups Fetches the cloudaeye security group under the given VPC (if exists)
- Create SecurityGroups Creates a cloudaeye security group under the given VPC (if not exists)
AWS ECS
- List TaskDefinition Families List all the task definition families
- Describe Task Definition Describes the given task definition
- List Tasks Lsits all avaialble tasks
- Describe Tasks Describes a given task
- Register Task Definition Register a new task definition for the metrics agent
- Run Task Runs a new task with the metrics agent task definition

AWS Permissions Required¶

To perform the above mentioned steps, the AWS SDK (boto3) needs the below set of permissions to be enabled

These permissions needs to be added to the AWS account used to deploy the agent. See how to configure an AWS account

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ecs:ListTaskDefinitionFamilies",
                "ecs:RegisterTaskDefinition",
                "ecs:DescribeTaskDefinition"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "iam:PassRole",
                "ecs:ListTasks",
                "ecs:DescribeTasks"
            ],
            "Resource": [
                "arn:aws:ecs:AWS_REGION:AWS_ACCOUNT_ID:container-instance/*",
                "arn:aws:ecs:AWS_REGION:AWS_ACCOUNT_ID:task/*",
                "arn:aws:iam::AWS_ACCOUNT_ID:role/*"
            ]
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ec2:CreateSecurityGroup"
            ],
            "Resource": [
                "arn:aws:ec2:AWS_REGION:AWS_ACCOUNT_ID:security-group/*",
                "arn:aws:ec2:AWS_REGION:AWS_ACCOUNT_ID:vpc/VPC_ID"
            ]
        }
    ]
}

AWS EKS(on EC2)¶

CloudAEye's EKS metrics agent script does the following:

Adds the agent as a daemonset on the EKS cluster using helmv3 chart

This steps needs helmv3 to be installed as a pre-requisite.
(Optional) Collects metrics of other AWS resources from Cloudwatch

To enable this Enable cloud services input needs to be provided.

How boto3 is used¶

To perform the above steps automatically, the agent script makes use of AWS boto3 python SDK. The AWS services accessed by the SDK are:

AWS CloudFormation

Applicable only if Enable cloud services is enabled
- Describe Stacks to check if a stack already exists for CloudAEyeServiceAccountStack
- Create Stack creates a stack CloudAEyeServiceAccountStack > This stack provisions a dedicated IAM user responsible to collect and ship metrics from the Cloudwatch to the log service
AWS IAM

Applicable only if Enable cloud services is enabled
- List Policy Versions Lists all available policies
- Get Policy Fetch the policy CloudAEyeServiceAccountPolicy (if exists)
- Create Policy Creates policy CloudAEyeServiceAccountPolicy (if not already exists)
- Delete Policy Deletes a given policy
- Get User Fetch user CloudAEyeServiceAccount (if exists)
- Create User Creates a programmatic user CloudAEyeServiceAccount (if not alrady exists)
- List AccessKeys List all keys available for the programmatic IAM user CloudAEyeServiceAccount
- Create AccessKey Create access key for the programmatic IAM user CloudAEyeServiceAccount
- Attach User Policy Attach user policy CloudAEyeServiceAccountPolicy to user CloudAEyeServiceAccount
AWS Cloudwatch

Applicable only if Enable cloud services is enabled
- Describe Log Groups Describe the given logs group

AWS Permissions Required¶

To perform the above mentioned steps, the AWS SDK (boto3) needs the below set of permissions to be enabled

These permissions needs to be added to the AWS account used to deploy the agent. See how to configure an AWS account

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "iam:GetUser",
                "iam:CreateUser",
                "iam:CreateAccessKey",
                "iam:ListAccessKeys"
                "iam:AttachUserPolicy",
                "iam:ListPolicyVersions",
                "iam:GetPolicy",
                "iam:CreatePolicy",
                "iam:DeletePolicy",
                "logs:DescribeLogGroups",
            ],
            "Resource": [
                "arn:aws:logs:AWS_REGION:AWS_ACCOUNT_ID:log-group:*",
                "arn:aws:iam::AWS_ACCOUNT_ID:user/CloudAEyeServiceAccount",
                "arn:aws:iam::AWS_ACCOUNT_ID:policy/CloudAEyeServiceAccountPolicy"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "cloudformation:CreateStack",
                "cloudformation:DescribeStacks"
            ],
            "Resource": "arn:aws:cloudformation:AWS_REGION:AWS_ACCOUNT_ID:stack/CloudAEyeServiceAccountStack/*"
        }
    ]
}