Skip to content

Setup Logs Streaming


Overview

This topic describes how user may setup logs streaming to send logs to CloudAEye SaaS.

⚓ Prerequisites

User is required to create a logs service.

Supported Environments

CloudAEye SaaS supports the following application environments:

Type Runtime Description / Cloud Services
Distributed cloud applications (typically follows microservices architecture) Docker with AWS EKS - Elastic Kubernetes Service, Kubernetes System Logs from AWS EKS
Containerized web applications Docker with AWS EC2 - Elastic Compute Cloud
Serverless containerized web applications Fargate with AWS ECS
Serverless functions Lambda

Streaming Logs

Install Agent on AWS EKS (Kubernetes)

CloudAEye SaaS console shows the instructions to setup agent for logs streaming. However, user will need to use caeops CLI to to set it up.

To view the instructions in console, select Services > Logs from the left navigation menu. User will see a list of logs services that are already created in the system. Click on the name of the logs service under Service Name column.

Under Agent instructions section, select EKS from the drop-down menu. The agent setup instructions for EKS will be shown. Similarly, select other choices from the drop-down menu to view instructions for other runtime environments such as EC2/Docker, ECS/Fargate.

User may run caeops logs install-agent command to setup the agent for logs streaming.

caeops logs install-agent --service-name demo-logs-service --cloud aws --source kubernetes --kubernetes-cluster-name demo-cluster  --app-name "demo-app"

where

  • --service-name: The logs service created in the create step mentioned in pre-requisites.
  • --kubernetes-cluster-name: Name of the kubernetes cluster running the applications
  • --app-name: An identification name to the application running on the kubernetes cluster.

CLI will run the following steps:

1. Download the script to run the agent. 
# This script deploys a helm chart. Skip this step if the script is already downloaded

   wget -O aws_kubernetes_agent.py \
     https://cae-data-collection-agent.s3.us-east-2.amazonaws.com/kubernetes/scripts/aws_kubernetes_agent.py

Do you want to execute the command ? (y/n) y

2. If no K8 agent is installed or active, run the below command to install a new K8 agent. 
# NOTE: If a K8 agent is already running for logs/metrics, skip this step and go to next step

   python3 aws_kubernetes_agent.py \
    --helm-repo "https://cae-data-collection-agent.s3.us-east-2.amazonaws.com/kubernetes/helm/cae-k8-agent/charts" \
    --enable-logs "yes" \
    --k8-cluster-name "demo-cluster" \
    --k8-exclude-logs-from-namespace "cloudaeye" \
    --k8-namespace "cloudaeye" \
    --cloud-env "aws" \
    --logs-destination "http" \
    --destination-http-url "{logs-destination-url}" \
    --app-name "demo-app" \
    --app-key "DA" \
    --user-key "{base64-encoded-access-key}" \
    --user-secret "{base64-encoded-access-key}" \
    --agent-mode "create" 

Do you want to execute the command ? (y/n) y

 .--. .-.                 .-. .--.  .--.
: .--': :                 : :: .; :: .--'
: :   : :   .--. .-..-. .-' ::    :: `;  .-..-. .--.
: :__ : :_ ' .; :: :; :' .; :: :: :: :__ : :; :' '_.'
`.__.'`.__;`.__.'`.__.'`.__.':_;:_;`.__.'`._. ;`.__.'
                                          .-. :
                                          `._.'

Install Agent with Helm

User may also run helm install on AWS EKS Kubernetes clusters to install the agent. Helm command is provided as instructions when logs service is created.

Here is an example:

helm install cae-k8-agent cae-k8-agent \
  --create-namespace \
  --namespace cloudaeye \
  --set namespace=cloudaeye \
  --set client.cloud_env=aws \
  --set client.app_name="demo-app" \
  --set client.app_key="DA" \
  --repo https://cae-data-collection-agent.s3.us-east-2.amazonaws.com/kubernetes/helm/cae-k8-agent/charts \
  --set user.key={base64-encoded-access-key} \
  --set user.secret={base64-encoded-access-secret-key} \
  --set client.cluster_name="demo-cluster" \
  --set logs.enabled=true \
  --set logs.source=kubernetes \
  --set "exclude_namespaces={cloudaeye}" \
  --set logs.destination=http \
  --set logs.url={logs-destination-url} \
  --set logs.kubernetes.controlPlane=false

Do you want to execute the helm command ?
(NOTE: Please make sure that you are connected to the right k8 cluster before running the helm command !) [y/n]: y
File association not found for extension .py
NAME: cae-k8-agent
LAST DEPLOYED: Wed Aug 25 12:20:38 2021
NAMESPACE: cloudaeye
STATUS: deployed
REVISION: 1
TEST SUITE: None

Kubernetes System Logs

CloudAEye supports collecting Kubernetes System Logs. Use --kubernetes-enable-system-logs option on caeops logs install-agent command to enable this feature.

Install Agent on Docker with AWS EC2

User may run caeops logs install-agent command with --source docker option to setup the agent for logs streaming on Docker with EC2 setup. Here is an example:

caeops logs install-agent --service-name demo-logs-service --cloud aws --source docker --docker-network "test" --app-name "demo-app"

where

  • --service-name is the logs service user created in create step.
  • --docker-network Name of the network on which the agent needs to run (Should be the same network on which the applications are running)
  • --app-name: An identification name to the application running on the kubernetes cluster.

The output of the above command looks like below:

Step 1
------
Download the script to run the agent. (This script deploys a docker container. Skip this step if the script is already downloaded)
←[36m    wget -O aws_docker_agent.py https://cae-data-collection-agent.s3.us-east-2.amazonaws.com/docker/scripts/aws_docker_agent.py

Do you want to execute the command ? (y/n)n


Step 2
------
Run the below command to install the agent
←[36m    python3 aws_docker_agent.py --agent-mode create --agent-type logs --docker-network test --logs-destination http --destination-http-url {logs-destination} --user-key {base64-encoded-access-key} --user-secret {base64-encoded-access-secret-key}

Do you want to execute the command ? (y/n)y

 .--. .-.                 .-. .--.  .--.
: .--': :                 : :: .; :: .--'
: :   : :   .--. .-..-. .-' ::    :: `;  .-..-. .--.
: :__ : :_ ' .; :: :; :' .; :: :: :: :__ : :; :' '_.'
`.__.'`.__;`.__.'`.__.'`.__.':_;:_;`.__.'`._. ;`.__.'
                                          .-. :
                                          `._.'


=======================================
Install Agent => Logs source: docker
=======================================

------------------------
Docker command generated
------------------------

docker run -d --name cae-logs-agent --network test  -e FLUENT_LOG_SOURCE_DOCKER=docker -e FLUENT_PLUGINS=docker -e FLUENT_UID=0 -e FLUENT_LOG_DESTINATION=http -e FLUENT_HTTP_ENDPOINT={logs-destination-url} -e CAE_CUSTOMER_ENV=aws -e APP_NAME="demo-app" -e CAE_ACCESS_KEY_ID={access-key} -e CAE_SECRET_ACCESS_KEY={access-secret-key} public.ecr.aws/b1r6v2a5/cae-logs-agent:latest

Do you want to run the docker container ? (NOTE: Please make sure that you have docker already installed!) [y/n] :