Setup Logs Streaming¶
Overview¶
This topic describes how user may setup logs streaming to send logs to CloudAEye SaaS.
Prerequisites¶
User is required to create a logs service.
Supported Environments¶
CloudAEye SaaS supports the following application environments:
Type | Runtime Description / Cloud Services |
---|---|
Distributed cloud applications (typically follows microservices architecture) | Docker with AWS EKS - Elastic Kubernetes Service, Kubernetes System Logs from AWS EKS |
Containerized web applications | Docker with AWS EC2 - Elastic Compute Cloud |
Serverless containerized web applications | Fargate with AWS ECS |
Serverless functions | Lambda |
Streaming Logs¶
Install Agent on AWS EKS (Kubernetes)¶
CloudAEye SaaS console shows the instructions to setup agent for logs streaming. However, user will need to use caeops
CLI to to set it up.
To view the instructions in console, select Services > Logs
from the left navigation menu. User will see a list of logs services that are already created in the system.
Click on the name of the logs service under Service Name
column.
Under Agent instructions
section, select EKS
from the drop-down menu. The agent setup instructions for EKS will be shown. Similarly, select other choices from the drop-down menu to view instructions for other runtime environments such as EC2/Docker
, ECS/Fargate
.
User may run caeops logs install-agent
command to setup the agent for logs streaming.
caeops logs install-agent --service-name demo-logs-service --cloud aws --source kubernetes --kubernetes-cluster-name demo-cluster --app-name "demo-app"
where
--service-name
: The logs service created in the create step mentioned in pre-requisites.--kubernetes-cluster-name
: Name of the kubernetes cluster running the applications--app-name
: An identification name to the application running on the kubernetes cluster.
CLI will run the following steps:
1. Download the script to run the agent.
# This script deploys a helm chart. Skip this step if the script is already downloaded
wget -O aws_kubernetes_agent.py \
https://cae-data-collection-agent.s3.us-east-2.amazonaws.com/kubernetes/scripts/aws_kubernetes_agent.py
Do you want to execute the command ? (y/n) y
2. If no K8 agent is installed or active, run the below command to install a new K8 agent.
# NOTE: If a K8 agent is already running for logs/metrics, skip this step and go to next step
python3 aws_kubernetes_agent.py \
--helm-repo "https://cae-data-collection-agent.s3.us-east-2.amazonaws.com/kubernetes/helm/cae-k8-agent/charts" \
--enable-logs "yes" \
--k8-cluster-name "demo-cluster" \
--k8-exclude-logs-from-namespace "cloudaeye" \
--k8-namespace "cloudaeye" \
--cloud-env "aws" \
--logs-destination "http" \
--destination-http-url "{logs-destination-url}" \
--app-name "demo-app" \
--app-key "DA" \
--user-key "{base64-encoded-access-key}" \
--user-secret "{base64-encoded-access-key}" \
--agent-mode "create"
Do you want to execute the command ? (y/n) y
.--. .-. .-. .--. .--.
: .--': : : :: .; :: .--'
: : : : .--. .-..-. .-' :: :: `; .-..-. .--.
: :__ : :_ ' .; :: :; :' .; :: :: :: :__ : :; :' '_.'
`.__.'`.__;`.__.'`.__.'`.__.':_;:_;`.__.'`._. ;`.__.'
.-. :
`._.'
Install Agent with Helm¶
User may also run helm install
on AWS EKS Kubernetes clusters to install the agent. Helm command is provided as instructions when logs service is created.
Here is an example:
helm install cae-k8-agent cae-k8-agent \
--create-namespace \
--namespace cloudaeye \
--set namespace=cloudaeye \
--set client.cloud_env=aws \
--set client.app_name="demo-app" \
--set client.app_key="DA" \
--repo https://cae-data-collection-agent.s3.us-east-2.amazonaws.com/kubernetes/helm/cae-k8-agent/charts \
--set user.key={base64-encoded-access-key} \
--set user.secret={base64-encoded-access-secret-key} \
--set client.cluster_name="demo-cluster" \
--set logs.enabled=true \
--set logs.source=kubernetes \
--set "exclude_namespaces={cloudaeye}" \
--set logs.destination=http \
--set logs.url={logs-destination-url} \
--set logs.kubernetes.controlPlane=false
Do you want to execute the helm command ?
(NOTE: Please make sure that you are connected to the right k8 cluster before running the helm command !) [y/n]: y
File association not found for extension .py
NAME: cae-k8-agent
LAST DEPLOYED: Wed Aug 25 12:20:38 2021
NAMESPACE: cloudaeye
STATUS: deployed
REVISION: 1
TEST SUITE: None
Kubernetes System Logs¶
CloudAEye supports collecting Kubernetes System Logs. Use --kubernetes-enable-system-logs
option on caeops logs install-agent
command to enable this feature.
Install Agent on Docker with AWS EC2¶
User may run caeops logs install-agent
command with --source docker
option to setup the agent for logs streaming on Docker with EC2 setup.
Here is an example:
caeops logs install-agent --service-name demo-logs-service --cloud aws --source docker --docker-network "test" --app-name "demo-app"
where
--service-name
is the logs service user created in create step.--docker-network
Name of the network on which the agent needs to run (Should be the same network on which the applications are running)--app-name
: An identification name to the application running on the kubernetes cluster.
The output of the above command looks like below:
Step 1
------
Download the script to run the agent. (This script deploys a docker container. Skip this step if the script is already downloaded)
←[36m wget -O aws_docker_agent.py https://cae-data-collection-agent.s3.us-east-2.amazonaws.com/docker/scripts/aws_docker_agent.py
Do you want to execute the command ? (y/n)n
Step 2
------
Run the below command to install the agent
←[36m python3 aws_docker_agent.py --agent-mode create --agent-type logs --docker-network test --logs-destination http --destination-http-url {logs-destination} --user-key {base64-encoded-access-key} --user-secret {base64-encoded-access-secret-key}
Do you want to execute the command ? (y/n)y
.--. .-. .-. .--. .--.
: .--': : : :: .; :: .--'
: : : : .--. .-..-. .-' :: :: `; .-..-. .--.
: :__ : :_ ' .; :: :; :' .; :: :: :: :__ : :; :' '_.'
`.__.'`.__;`.__.'`.__.'`.__.':_;:_;`.__.'`._. ;`.__.'
.-. :
`._.'
=======================================
Install Agent => Logs source: docker
=======================================
------------------------
Docker command generated
------------------------
docker run -d --name cae-logs-agent --network test -e FLUENT_LOG_SOURCE_DOCKER=docker -e FLUENT_PLUGINS=docker -e FLUENT_UID=0 -e FLUENT_LOG_DESTINATION=http -e FLUENT_HTTP_ENDPOINT={logs-destination-url} -e CAE_CUSTOMER_ENV=aws -e APP_NAME="demo-app" -e CAE_ACCESS_KEY_ID={access-key} -e CAE_SECRET_ACCESS_KEY={access-secret-key} public.ecr.aws/b1r6v2a5/cae-logs-agent:latest
Do you want to run the docker container ? (NOTE: Please make sure that you have docker already installed!) [y/n] :