Setup Logs Analyzer¶

Overview¶

Distributed cloud applications usually generate logs (e.g. states, events) that are frequently used in maintenance and diagnosis of the systems. Inspecting logs has become a norm during production failure. Logs plays a critical role during investigating a production incident of a cloud applications since it is very hard to use a debugging tool in this type of envrionment.

Static dashboards with human-generated thresholds do not scale for modern cloud applications

Logs Analyzer is an AI-powered managed service that surfaces anomalous logs from applications by providing actionable operational insights. It intelligently analyzes the data in near real time using machine learning (ML) and deep learning (DL) models and can detect errors or any anomalous activities in user applications within minutes of their occurrence. This helps to reduce MTTD (mean time to detect).

This topic describes how to setup the AI powered logs analyzers for realtime anomaly detection.

Prerequisites¶

User must setup a logs service and metrics service for the same application.
Both logs and metrics services must be added to a service group.

To ensure that there is enough data for training the AI/ML model(s), we recommend creating the logs analyzer few (1-3) days after the creation of the corresponding logs & metrics services.

Why?¶

Distributed cloud applications can produce large amount (e.g. Terabytes) of logs per day. It is very hard to analyze so much volume of logs manually
A well-deployed application that operates normally most of the time will produce logs that are normal. Only a small percentage of logs would indicate problems and are hard to pin point.
Modern cloud applications are changed on a regular (e.g. daily or weekly) basis. It is very hard to keep track of the applications using a static approach
It is very hard to model seasonality with static alerts. A system with online adaptive learning algorithms is required.
Downtime is expensive. Having the ability to detect incidents in a timely manner saves enterprises money and reputation

Distributed system architectures increase the need for observability because such architectures can fail due to interaction between multiple systems. -- Gartner

How it Works?¶

CloudAEye offers both unsupervised and semi-supervised models.

In case of unsupervised model, the model learns log patterns from normal execution and can detect an anomaly when the log pattern deviates.
In case of semi-supervised model, the model is given small amount of labeled data so that it can learn both normal and abnormal patterns.

Anomaly Score¶

Our models rank the anomalies detected based on the significance of an anomaly score. An anomaly score usually represents the confidence level of the model about the likelyhood that the detected incident is an anomaly.

CloudAEye uses the following rank categories to communicate the likelyhood of an anomaly:

Anomaly Score	Confidence Level
0-25	`low`
25-50	`medium`
50-75	`high`
75-100	`very high`

Setup¶

Create a New Logs Analyzer¶

ConsoleCLI

From left navigation menu, select Services > Logs Analyzer. A list of logs analyzer services that are already created will be shown. The table will be empty if there are no logs analyzer services being created in the system.

To create a new logs analyzer service, click on CREATE on the top right corner. A new form will appear under Logs Analyzer > Create.

Provide the following informaiton in the form:

Name: Name of the logs analyzer service. This is usually an alpha-numeric string. For example, orders-app-logs-analyzer.
Data source (logs): Pick the Logs Service this service will be analyzing. A data pipeline will be created from the logs service. All logs data will then be analyzed by AI models.
Data source (metrics): Pick the Metrics Service from the same application. Analyzer service will leverage the metrics to correlate and identify anomalies.

Click SUBMIT to create the logs analyzer service.

User may use the command described below to create a logs analyzer service.

caeops logs-analyzers create --name=demo-logs-analyzer --data-sources=[{logs=demo-logs-service},{metrics=demo-metrics-service}]

where

--data-sources: - logs : points to an ElasticSearch based logs service - metrics : points to an Prometheus based metrics service

This will initiate training of AI/ML models and deploy them for realtime logs analysis. A data-pipeline will be created between Elasticsearch and AI/ML models.

Output from the CLI command may look like the following:

{
  "serviceName": "demo-logs-analyzer",
  "serviceType": "logs-analyzer",
  "groupName": "demo-grp",
  "dataSources": {
    "logs": "demo-logs-service",
    "metrics": "demo-metrics-service"
  },
  "createdAt": 1629949067277,
  "updatedAt": 1629949067277
}

List All Logs Analyzer(s)¶

ConsoleCLI

From left navigation menu, select Services > Logs Analyzer. A list of logs analyzer services that are already created will be shown. The table will be empty if there are no logs analyzer services being created in the system.

Click on a specific service name link under Service Name column to see details of a Logs Analyzer service.

The following information is shown in the details page:

Name - Name of the logs analyzer service.
Date created - Date when the logs analyzer service was created.
Date updated - Date when the logs analyzer service was last updated.
Group name - Name of the Service Groups this analyzer is analyzing.
Data Source - Logs and metrics services associated with this analyzer.
Dashboard - Logs analyzer dashboard. Click OPEN to see the dashboard. This shows the logs anomalies associated with the logs service.

User may use the command shown below to list all the created log analyzers

caeops logs-analyzers list

The output from the command may look like the following:

[
  {
    "serviceName": "demo-logs-analyzer",
    "serviceType": "logs-analyzer",
    "groupName": "demo-grp",
    "dataSources": {
      "logs": "demo-logs-service",
      "metrics": "demo-metrics-service"
    },
    "createdAt": 1629949067277,
    "updatedAt": 1629949067277
  }
]

Delete a Logs Analyzer¶

ConsoleCLI

From left navigation menu, select Services > Logs Analyzer. A list of logs analyzer services that are already created will be shown. The table will be empty if there are no logs analyzer services being created in the system.

Click on X button under Actions column to delete a specific Logs Analyzer. A confirmation windown will be shown.

Click CONFIRM to delete the Logs Analyzer.

User may use the command shown below to delete a particular logs analyzer.

caeops logs-analyzers delete --name=demo-logs-analyzer