Skip to content

Setup Logs Parsing


Overview

CloudAEye supports sophisticated logs parsing capabilities. This topic describes how user may setup logs parsing.

⚓ Prerequisites

User is required to create a logs service.

Logs Parsing

CloudAEye leverages popular opensource technologies such as logstash to deliver sophisticated logs parsing with end-to-end integrations. Here are the logs parsing features currently supported by CloudAEye logs service.

Grok Filter based Logs Parsing

Create Parsing Rule

User may use caeops logs create-parsing-rule command. Here is an example:

caeops logs create-parsing-rule \
  --name controller-pattern \
  --service-name demo-logs-service \
  --filter '{\"pattern\":\"(?<ip>\\d{1,3}.\\d{1,3}.\\d{1,3}.\\d{1,3}) - - \\[(?<date>\\d{2}\\\/\\w{3}\\\/\\d{4}:\\d{2}:\\d{2}:\\d{2} \\+\\d{4})\\] \\\"(?<method>\\b\\w+\\b) \\\/(?<service>\\b\\w+\\b?) (?<http>.*?)\\\" (?<code>\\d{3}) (?<bytes>\\d{3}) \\\"-\\\" \\\"(?<path>.*?)\\\" (?<count>\\d{3}) (?<message>.*)\",\"target\":\"controller_meta\"}'
Here --service-name is the logs service user created in create step.

The output from the command may look like the following:

{
  "createdAt": 1629913360323,
  "updatedAt": 1629916772245,
  "name": "controller-pattern",
  "pluginName": "grok",
  "rule": {
    "pattern": "(?<ip>\\d{1,3}.\\d{1,3}.\\d{1,3}.\\d{1,3}) - - \\[(?<date>\\d{2}\\/\\w{3}\\/\\d{4}:\\d{2}:\\d{2}:\\d{2} \\+\\d{4})\\] \"(?<method>\\b\\w+\\b) \\/(?<service>\\b\\w+\\b?) (?<http>.*?)\" (?<code>\\d{3}) (?<bytes>\\d{3}) \"-\" \"(?<path>.*?)\" (?<count>\\d{3}) (?<message>.*)",
    "target": "controller_meta"
  },
  "active": false
}

List Parsing Rules

User may use caeops logs list-parsing-rules command. Here is an example:

caeops logs list-parsing-rules --service-name demo-logs-service

Here --service-name is the logs service user created in create step.

The output from the command may look like the following:

[
  {
    "createdAt": 1629913360323,
    "updatedAt": 1629916772245,
    "name": "controller-pattern",
    "pluginName": "grok",
    "rule": {
      "pattern": "(?<ip>\\d{1,3}.\\d{1,3}.\\d{1,3}.\\d{1,3}) - - \\[(?<date>\\d{2}\\/\\w{3}\\/\\d{4}:\\d{2}:\\d{2}:\\d{2} \\+\\d{4})\\] \"(?<method>\\b\\w+\\b) \\/(?<service>\\b\\w+\\b?) (?<http>.*?)\" (?<code>\\d{3}) (?<bytes>\\d{3}) \"-\" \"(?<path>.*?)\" (?<count>\\d{3}) (?<message>.*)",
      "target": "controller_meta"
    },
    "active": false
  }
]

Enable Parsing Rule

User may enable an existing rule using caeops logs enable-parsing-rule command. Here is an example:

caeops logs enable-parsing-rule --name controller-pattern --service-name demo-logs-service

Here --service-name is the logs service user created in create step.

The output from the command may look like the following:

{
  "createdAt": 1629913360323,
  "updatedAt": 1629916772245,
  "name": "controller-pattern",
  "pluginName": "grok",
  "rule": {
    "pattern": "(?<ip>\\d{1,3}.\\d{1,3}.\\d{1,3}.\\d{1,3}) - - \\[(?<date>\\d{2}\\/\\w{3}\\/\\d{4}:\\d{2}:\\d{2}:\\d{2} \\+\\d{4})\\] \"(?<method>\\b\\w+\\b) \\/(?<service>\\b\\w+\\b?) (?<http>.*?)\" (?<code>\\d{3}) (?<bytes>\\d{3}) \"-\" \"(?<path>.*?)\" (?<count>\\d{3}) (?<message>.*)",
    "target": "controller_meta"
  },
  "active": true
}

Disable Parsing Rule

User may disable an exising rule using caeops logs disable-parsing-rule command. Here is an example:

caeops logs disable-parsing-rule --name controller-pattern --service-name demo-logs-service

Here --service-name is the logs service user created in create step.

The output from the command may look like the following:

{
  "createdAt": 1629913360323,
  "updatedAt": 1629916772245,
  "name": "controller-pattern",
  "pluginName": "grok",
  "rule": {
    "pattern": "(?<ip>\\d{1,3}.\\d{1,3}.\\d{1,3}.\\d{1,3}) - - \\[(?<date>\\d{2}\\/\\w{3}\\/\\d{4}:\\d{2}:\\d{2}:\\d{2} \\+\\d{4})\\] \"(?<method>\\b\\w+\\b) \\/(?<service>\\b\\w+\\b?) (?<http>.*?)\" (?<code>\\d{3}) (?<bytes>\\d{3}) \"-\" \"(?<path>.*?)\" (?<count>\\d{3}) (?<message>.*)",
    "target": "controller_meta"
  },
  "active": false
}