Skip to content

caeops > logs

install-agent

Description

This command helps you setup an agent that collects logs from any source (supported by CloudAEye) running on a given cloud env (supported by CloudAEye) and ships them to the provided logs service endpoint

Run caeops logs install-agent --help for more help.

Synopsis

  install-agent
--service-name [value]
--cloud [value]
--source [value]
--app-name [value]
[--kubernetes-cluster-name [value]]
[--kubernetes-enable-system-logs [value]]
[--docker-network [value]]
[--ecs-cluster-name [value]]

Options

--service-name (string)

Name of the logs service

--cloud (string)

The type of cloud service providers. Supported providers : aws

--source (string)

The type of cloud source. Supported resources : kubernetes | docker | ecs-fargate

--app-name (string)

A representative name for the current app sending the logs

--kubernetes-cluster-name (string)

Name of the kubernetes cluster where the agent needs to run (for source='kubernetes')

--kubernetes-enable-system-logs (bool)

Determines whether or not to collect system level logs for the cluster (for source='kubernetes')

--docker-network (string)

Specifies the docker network on which the agent needs to run (for source='docker')

--ecs-cluster-name (string)

Name of the ECS cluster where the agent needs to run (for source='ecs')

Examples

Example 1

  • The following logs install-agent example generates commands to install an agent for a kubernetes source on aws cloud
caeops logs install-agent
        --service-name mylogs --cloud aws --source kubernetes --app-name "Test App"
        --kubernetes-cluster-name test-cluster --kubernetes-enable-system-logs true
Output

Instructions -> (string)

Step 1
------
Download the script to run the agent. (This script deploys a helm chart. Skip this step if the script is already downloaded)
    wget -O aws_kubernetes_agent.py https://cae-data-collection-agent.s3.us-east-2.amazonaws.com/kubernetes/scripts/aws_kubernetes_agent.py

Do you want to execute the command ? (y/n)

Step 2
------
If no K8 agent is installed or active, run the below command to install a new K8 agent. (NOTE: If a K8 agent is already running for logs/metrics, skip this step and go to next step)
    python3 aws_kubernetes_agent.py --helm-repo "https://cae-data-collection-agent.s3.us-east-2.amazonaws.com/kubernetes/helm/cae-k8-agent/charts" --enable-logs "yes" --k8-cluster-name "test-cluster" --k8-exclude-logs-from-namespace "cloudaeye" --k8-namespace "cloudaeye" --cloud-env "aws" --logs-destination "http" --destination-http-url "http://endpoint.com" --app-name "Test App" --app-key "TA" --user-key somekey --user-secret somesecret --agent-mode "create"

Do you want to execute the command ? (y/n)

If a K8 agent is already running, run the below command to update the existing K8 agent. (NOTE: Skip this step if a new agent was installed in previous step)
   python3 aws_kubernetes_agent.py --helm-repo "https://cae-data-collection-agent.s3.us-east-2.amazonaws.com/kubernetes/helm/cae-k8-agent/charts" --enable-logs "yes" --k8-cluster-name "test-cluster" --k8-exclude-logs-from-namespace "cloudaeye" --k8-namespace "cloudaeye" --cloud-env "aws" --logs-destination "http" --destination-http-url "http://endpoint.com" --app-name "Test App" --app-key "TA" --user-key somekey --user-secret somesecret --agent-mode "update"

Example 2

  • The following logs install-agent example generates commands to install an agent for a docker source on aws cloud
caeops logs install-agent --service-name mylogs --cloud aws --source docker --app-name "Dev App" --docker-network test
Output

Instructions -> (string)

Step 1
------
Download the script to run the agent. (This script deploys a docker container. Skip this step if the script is already downloaded)
    wget -O aws_docker_agent.py https://cae-data-collection-agent.s3.us-east-2.amazonaws.com/docker/scripts/aws_docker_agent.py

Do you want to execute the command ? (y/n)

Step 2
------
Run the below command to install the agent
   python3 aws_docker_agent.py --agent-mode create --agent-type logs --docker-network test --cloud-env aws --logs-destination http --destination-http-url http://endpoint.com --app-name "Dev App" --app-key DA --user-key somekey --user-secret somesecret

Do you want to execute the command ? (y/n)

Step 3
------
For Docker Containers: Restart your app containers with an additional argument --log-driver=fluentd to use fluentd as log driver and ship logs to destination. See example below
docker stop my-container
docker rm my-container
docker run --log-driver=fluentd --name my-container nginx --network test


Step 4
------
For Docker Compose: Update your docker-compose.yaml to set logging.driver=fluentd to use fluentd as log driver driver and ship logs to destination. See example below
docker-compose down

    version: "3"
    services:
    nginx:
    image: nginx
    container_name: nginx
    logging:
      driver: fluentd
    networks:
      - web
    networks:
    web:
      external: true
      name: --network test

Example 3

  • The following logs install-agent example generates commands to install an agent for a ecs-fargate source on aws cloud
caeops logs install-agent --service-name mylogs --cloud aws --source ecs-fargate --app-name "Dev App" --ecs-cluster-name test-cluster
Output

Instructions -> (string)

Step 1
------
1.0> Copy and save the generated task definition template to a local file. This template runs CloudAEye"s agent as a side-car with your container.
Edit the template to replace the placeholders with your app details

{
  "family": "sample-definition",
  "taskRoleArn": "${TASK_ROLE_ARN}",
  "executionRoleArn": "${EXECUTION_ROLE_ARN}",
  "networkMode": "awsvpc",
  "containerDefinitions": [
    {
      "essential": true,
      "image": "public.ecr.aws/cloudaeye/cae-logs-agent:latest",
      "name": "log_router",
      "firelensConfiguration": {
        "type": "fluentd"
      },
      "environment": [
        {
          "name": "FLUENTD_CONF",
          "value": "firelens.conf"
        },
        {
          "name": "FLUENT_LOG_SOURCE_ECS_FARGATE_LOCAL",
          "value": "ecs_fargate_local"
        },
        {
          "name": "FLUENT_PLUGINS",
          "value": "ecs_fargate_local"
        },
        {
          "name": "FLUENT_UID",
          "value": "0"
        },
        {
          "name": "FLUENT_LOG_DESTINATION",
          "value": "http"
        },
        {
          "name": "FLUENT_HTTP_ENDPOINT",
          "value": "http://yourendpoint.com"
        },
        {
          "name": "CAE_CUSTOMER_ENV",
          "value": "aws"
        },
        {
          "name": "APP_NAME",
          "value": "Demo App"
        },
        {
          "name": "APP_KEY",
          "value": "DA"
        },
        {
          "name": "CAE_ACCESS_KEY_ID",
            "value": "somekey"
        },
        {
          "name": "CAE_SECRET_ACCESS_KEY",
          "value": "somesecret"
        }
      ]
    },
    {
      "name": "${YOUR_CONTAINER_NAME}",
      "image": "${YOUR_APP_IMAGE}",
      "logConfiguration": {
        "logDriver": "awsfirelens"
      }
    }
  ],
  "requiresCompatibilities": [
    "FARGATE"
  ]
}


Step 2
------
2.0> Now create/update your task definition (Make sure you have necessary AWS permissions)
aws ecs register-task-definition --cli-input-json file://TEMPLATE_DEFINITION_FILE.json


2.2> Run/update task with the above definition template (Make sure you have necessary AWS permissions)
aws ecs run-task --cluster todo-app --task-definition TASK_DEFINITION_NAME --network-configuration awsvpcConfiguration={subnets=[""],securityGroups=[""],assignPublicIp=""}
Back to top