Streaming Logs: Docker on AWS EC2

---

Overview

This topic describes how user may setup log streaming for containerized web applications running on AWS EC2.

Prerequisites

• An EC2 instance. To learn how to setup an EC2 instance please click here.

Streaming Logs

Install Agent - Docker on AWS EC2

Console

CloudAEye SaaS console shows the instructions to setup agent for logs streaming.

To view the instructions in console, select Logs Services from the left navigation menu. User will see a list of logs services that are already created in the system. Click on the name of the logs service under Service Name column.

Configure App for Log Collection

Under Configure App for Log Collection section, select Docker from the drop-down menu.

Enter the following information:

• App Name: Name of your application.
• Network: Name of the network on which the docker containers are running.
• Instance IDs: EC2 instance IDs.
• Instance Tags: Tags to select the instance IDs.

Click Create button. The agent setup instructions for Docker will be shown.
You may also click on the View Commands icon on the app to see the agent setup instructions.

Step 1: Download the script

You will see a command similar to the following:

curl -o aws_docker_agent.py https://cae-data-collection-agent.s3.us-east-2.amazonaws.com/docker/scripts/aws_docker_agent.py

Run this command on AWS CloudShell. You will get this output

Step 2: Install boto3

Run the following command on AWS CloudStell. You will get this output

pip3 install boto3 pyyaml requests

Step 3: Install Agent

python3 aws_docker_agent.py \
   --agent-mode create \
   --agent-type logs \
   --docker-network test \ 
   --instance-ids i-abc,i-xyz \
   --instance-tags 'evn=dev' \
   --cloud-env aws \
   --logs-destination http \
   --destination-http-url ${deatination-url} \
   --app-name 'My App' \
   --app-key MA \
   --user-key ${user-key} \
   --user-secret ${user-secret} \
   --execution-type remote \
   --auto-execute

Run a command similar to the following on AWS CloudShell. You will get this output

The following commands will have to be run in the EC2 instance

For Docker Compose

Set logging.driver=fluentd in your docker-compose.yaml if you are using docker compose. This will set fluentd as the log driver.

The docker-compose.yaml configuration will look like the following:

version: '3'
services:
    nginx:
        image: nginx
        container_name: sample-container
        logging:
            driver: fluentd
        networks:
            - mynetwork
networks:
    mynetwork:
        external: true
        name: test

For Docker Container

Restart your application docker container with an additional argumant --log-driver=fluentd to use fluentd as the log driver.

docker stop sample-container
docker rm sample-container
docker run --log-driver=fluentd --name sample-container nginx --network test

Agent Details

Please refer to agent description under Troubleshooting section for details.

Troubleshooting

Please refer to log streaming description under Troubleshooting section for details.