Skip to content

Logs Agent Script


Overview

Logs agent is usually an executable script/command responsible to collect logs from the required services running on the cloud and stream them to the respective logs service created on CloudAEye.

This topic provides complete details about the the logs agent script deployed on your cloud environment to collect logs from the selected services.

⚓ Prerequisites

CloudAEye makes use of python to automate the process of deploying an agent on to your cloud environment.

  • User must have python3 installed on the system

How it Works?

AWS Lambda

CloudAEye's lambda logs agent script does the following:

  1. Adds the agent as an extension and updates all the lambda functions provided using the Function names input

    In case of * as input, all lambda functions are selected.

  2. Adds the required environment variables to support the extension

How boto3 is used

To perform the above steps automatically, the agent script makes use of AWS boto3 python SDK. The AWS services accessed by the SDK are:

AWS Permissions Required

To perform the above mentioned steps, the AWS SDK (boto3) needs the below set of permissions to be enabled

These permissions needs to be added to the AWS account used to deploy the agent. See how to configure an AWS account

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "lambda:UpdateFunctionConfiguration",
                "lambda:GetFunctionConfiguration",
                "lambda:PublishVersion"
            ],
            "Resource": "arn:aws:lambda:AWS_REGION:AWS_ACCOUNT_ID:function:*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "lambda:GetLayerVersion",
            "Resource": "arn:aws:lambda:AWS_REGION:AWS_ACCOUNT_ID:layer:*:*"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "lambda:ListFunctions",
            "Resource": "*"
        }
    ]
}

AWS ECS(Fargate)

CloudAEye's ECS logs agent script does the following:

  1. Adds the agent as a side car container and updates all the task definitions used by the ECS services provided using the Service names input

    In case of * as input, all services running on the cluster are selected.

  2. (Optional) Updates the services with the new task definition registered in above step

    To enable this Update services needs to be selected.

How boto3 is used

To perform the above steps automatically, the agent script makes use of AWS boto3 python SDK. The AWS services accessed by the SDK are:

AWS Permissions Required

To perform the above mentioned steps, the AWS SDK (boto3) needs the below set of permissions to be enabled

These permissions needs to be added to the AWS account used to deploy the agent. See how to configure an AWS account

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ecs:ListServices",
                "ecs:ListTaskDefinitionFamilies",
                "ecs:RegisterTaskDefinition",
                "ecs:DescribeTaskDefinition"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "ecs:UpdateService",
                "iam:PassRole",
                "ecs:ListTasks",
                "ecs:DescribeServices",
                "ecs:DescribeTasks"
            ],
            "Resource": [
                "arn:aws:ecs:AWS_REGION:AWS_ACCOUNT_ID:service/*",
                "arn:aws:ecs:AWS_REGION:AWS_ACCOUNT_ID:container-instance/*",
                "arn:aws:ecs:AWS_REGION:AWS_ACCOUNT_ID:task/*",
                "arn:aws:iam::AWS_ACCOUNT_ID:role/*"
            ]
        }
    ]
}

AWS EKS(on EC2)

CloudAEye's EKS logs agent script does the following:

  1. Adds the agent as a daemonset on the EKS cluster using helmv3 chart

    This steps needs helmv3 to be installed as a pre-requisite.

  2. (Optional) Enables the control plane logging(if not already enabled) and collects the corresponding logs from AWS Cloudwatch

    To enable this Enable System Logs needs to be selected.

How boto3 is used

To perform the above steps automatically, the agent script makes use of AWS boto3 python SDK. The AWS services accessed by the SDK are:

  • AWS CloudFormation

    Applicable only if Enable system logs is enabled

    • Describe Stacks to check if a stack already exists for CloudAEyeServiceAccountStack
    • Create Stack creates a stack CloudAEyeServiceAccountStack > This stack provisions a dedicated IAM user responsible to collect and ship logs from the EKS Cloudwatch to the log service
  • AWS IAM

    Applicable only if Enable system logs is enabled

    • List Policy Versions Lists all available policies
    • Get Policy Fetch the policy CloudAEyeServiceAccountPolicy (if exists)
    • Create Policy Creates policy CloudAEyeServiceAccountPolicy (if not already exists)
    • Delete Policy Deletes a given policy
    • Get User Fetch user CloudAEyeServiceAccount (if exists)
    • Create User Creates a programmatic user CloudAEyeServiceAccount (if not alrady exists)
    • List AccessKeys List all keys available for the programmatic IAM user CloudAEyeServiceAccount
    • Create AccessKey Create access key for the programmatic IAM user CloudAEyeServiceAccount
    • Attach User Policy Attach user policy CloudAEyeServiceAccountPolicy to user CloudAEyeServiceAccount
  • AWS EKS

    Applicable only if Enable system logs is enabled

  • AWS Cloudwatch

    Applicable only if Enable system logs is enabled

AWS Permissions Required

To perform the above mentioned steps, the AWS SDK (boto3) needs the below set of permissions to be enabled

These permissions needs to be added to the AWS account used to deploy the agent. See how to configure an AWS account

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "eks:UpdateClusterConfig",
                "eks:DescribeUpdate",
                "eks:DescribeCluster",
                "iam:GetUser",
                "iam:CreateUser",
                "iam:CreateAccessKey",
                "iam:ListAccessKeys"
                "iam:AttachUserPolicy",
                "iam:ListPolicyVersions",
                "iam:GetPolicy",
                "iam:CreatePolicy",
                "iam:DeletePolicy",
                "logs:DescribeLogGroups",
            ],
            "Resource": [
                "arn:aws:logs:AWS_REGION:AWS_ACCOUNT_ID:log-group:*",
                "arn:aws:eks:AWS_REGION:AWS_ACCOUNT_ID:cluster/EKS_CLUSTER_NAME",
                "arn:aws:iam::AWS_ACCOUNT_ID:user/CloudAEyeServiceAccount",
                "arn:aws:iam::AWS_ACCOUNT_ID:policy/CloudAEyeServiceAccountPolicy"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "cloudformation:CreateStack",
                "cloudformation:DescribeStacks"
            ],
            "Resource": "arn:aws:cloudformation:AWS_REGION:AWS_ACCOUNT_ID:stack/CloudAEyeServiceAccountStack/*"
        }
    ]
}