Streaming Logs: Kubernetes (AWS EKS)

---

Overview

This topic describes how to setup log streaming for a distributed cloud application running on AWS EKS (Kubernetes).

Streaming Logs

Install Agent on AWS EKS (Kubernetes)

Console

CloudAEye SaaS console shows the instructions to setup agent for logs streaming.

To view the instructions in console, select Logs Services from the left navigation menu. User will see a list of logs services that are already created in the system. Click on the name of the logs service under Service Name column.

Configure App for Log Collection

Under Configure App for Log Collection section, select EKS from the drop-down menu.

Please enter the following information:

• App Name: Name of your application.
• Cluster Name: Name of your kubernetes cluster.
• AWS Region: Name of AWS regsion where the EKS cluster is running.
• Enable System Logs: Enable this check box to collect Kubernetes system (control panel) logs from CloudWatch.

Press the Create button. The agent setup instructions for EKS will be shown. You may also click on the View Commands icon on the app to see the agent setup instructions.

Step 1: Download the script

You will see a command similar to the following:

curl -o aws_kubernetes_agent.py https://cae-data-collection-agent.s3.us-east-2.amazonaws.com/kubernetes/scripts/aws_kubernetes_agent.py

Run this command on AWS CloudShell.

Step 2: Configure kubectl

Configure kubectl so that you can connect to your EKS cluster.

Run a command similar to the following on AWS CloudShell.

aws eks --region us-east-2 update-kubeconfig --name my-cluster

Step 3: Install helm

Run a command similar to the following on AWS CloudShell.

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 chmod 700 get_helm.sh export VERIFY_CHECKSUM=false ./get_helm.sh

Step 4: Install boto3

Run the following command on AWS CloudStell.

pip3 install boto3 

Install Agent: If no agent is already installed

Run a command similar to the following on AWS CloudShell.

python3 aws_kubernetes_agent.py \
   --helm-repo 'https://cae-data-collection-agent.s3.us-east-2.amazonaws.com/kubernetes/helm/cae-k8-agent/charts' \
   --enable-logs 'yes' \
   --k8-cluster-name 'my-cluster' \
   --k8-exclude-logs-from-namespace 'cloudaeye' \
   --k8-enable-control-plane-logs 'on' \
   --k8-namespace 'cloudaeye' \
   --cloud-env 'aws' \
   --logs-destination 'http' \
   --destination-http-url ${deatination-url} \
   --app-name 'My App' \
   --app-key 'MA' \
   --user-key ${user-key} \
   --user-secret ${user-secret} \
   --agent-mode 'create'

Update Agent: If a K8 agent is already running

Run a command similar to the following on AWS CloudShell.

python3 aws_kubernetes_agent.py \
   --helm-repo 'https://cae-data-collection-agent.s3.us-east-2.amazonaws.com/kubernetes/helm/cae-k8-agent/charts' \
   --enable-logs 'yes' \
   --k8-cluster-name 'my-cluster' \
   --k8-exclude-logs-from-namespace 'cloudaeye' \
   --k8-enable-control-plane-logs 'on' \
   --k8-namespace 'cloudaeye' \
   --cloud-env 'aws' \
   --logs-destination 'http' \
   --destination-http-url ${deatination-url} \
   --app-name 'My App' \
   --app-key 'MA' \
   --user-key ${user-key} \
   --user-secret ${user-secret} \
   --agent-mode 'update'

Agent Details

Please refer to agent description under Troubleshooting section for details.

Troubleshooting

Please refer to log streaming description under Troubleshooting section for details.