Setup Logs Parsing¶
Overview¶
CloudAEye supports sophisticated logs parsing capabilities. This topic describes how user may setup logs parsing.
Prerequisites¶
User is required to create a logs service.
Logs Parsing¶
CloudAEye leverages popular opensource technologies such as logstash to deliver sophisticated logs parsing with end-to-end integrations. Here are the logs parsing features currently supported by CloudAEye logs service.
Grok Filter based Logs Parsing¶
Create Parsing Rule¶
User may use caeops logs create-parsing-rule
command. Here is an example:
caeops logs create-parsing-rule \
--name controller-pattern \
--service-name demo-logs-service \
--filter '{\"pattern\":\"(?<ip>\\d{1,3}.\\d{1,3}.\\d{1,3}.\\d{1,3}) - - \\[(?<date>\\d{2}\\\/\\w{3}\\\/\\d{4}:\\d{2}:\\d{2}:\\d{2} \\+\\d{4})\\] \\\"(?<method>\\b\\w+\\b) \\\/(?<service>\\b\\w+\\b?) (?<http>.*?)\\\" (?<code>\\d{3}) (?<bytes>\\d{3}) \\\"-\\\" \\\"(?<path>.*?)\\\" (?<count>\\d{3}) (?<message>.*)\",\"target\":\"controller_meta\"}'
--service-name
is the logs service user created in create step.
The output from the command may look like the following:
{
"createdAt": 1629913360323,
"updatedAt": 1629916772245,
"name": "controller-pattern",
"pluginName": "grok",
"rule": {
"pattern": "(?<ip>\\d{1,3}.\\d{1,3}.\\d{1,3}.\\d{1,3}) - - \\[(?<date>\\d{2}\\/\\w{3}\\/\\d{4}:\\d{2}:\\d{2}:\\d{2} \\+\\d{4})\\] \"(?<method>\\b\\w+\\b) \\/(?<service>\\b\\w+\\b?) (?<http>.*?)\" (?<code>\\d{3}) (?<bytes>\\d{3}) \"-\" \"(?<path>.*?)\" (?<count>\\d{3}) (?<message>.*)",
"target": "controller_meta"
},
"active": false
}
List Parsing Rules¶
User may use caeops logs list-parsing-rules
command. Here is an example:
caeops logs list-parsing-rules --service-name demo-logs-service
Here --service-name
is the logs service user created in create step.
The output from the command may look like the following:
[
{
"createdAt": 1629913360323,
"updatedAt": 1629916772245,
"name": "controller-pattern",
"pluginName": "grok",
"rule": {
"pattern": "(?<ip>\\d{1,3}.\\d{1,3}.\\d{1,3}.\\d{1,3}) - - \\[(?<date>\\d{2}\\/\\w{3}\\/\\d{4}:\\d{2}:\\d{2}:\\d{2} \\+\\d{4})\\] \"(?<method>\\b\\w+\\b) \\/(?<service>\\b\\w+\\b?) (?<http>.*?)\" (?<code>\\d{3}) (?<bytes>\\d{3}) \"-\" \"(?<path>.*?)\" (?<count>\\d{3}) (?<message>.*)",
"target": "controller_meta"
},
"active": false
}
]
Enable Parsing Rule¶
User may enable an existing rule using caeops logs enable-parsing-rule
command. Here is an example:
caeops logs enable-parsing-rule --name controller-pattern --service-name demo-logs-service
Here --service-name
is the logs service user created in create step.
The output from the command may look like the following:
{
"createdAt": 1629913360323,
"updatedAt": 1629916772245,
"name": "controller-pattern",
"pluginName": "grok",
"rule": {
"pattern": "(?<ip>\\d{1,3}.\\d{1,3}.\\d{1,3}.\\d{1,3}) - - \\[(?<date>\\d{2}\\/\\w{3}\\/\\d{4}:\\d{2}:\\d{2}:\\d{2} \\+\\d{4})\\] \"(?<method>\\b\\w+\\b) \\/(?<service>\\b\\w+\\b?) (?<http>.*?)\" (?<code>\\d{3}) (?<bytes>\\d{3}) \"-\" \"(?<path>.*?)\" (?<count>\\d{3}) (?<message>.*)",
"target": "controller_meta"
},
"active": true
}
Disable Parsing Rule¶
User may disable an exising rule using caeops logs disable-parsing-rule
command. Here is an example:
caeops logs disable-parsing-rule --name controller-pattern --service-name demo-logs-service
Here --service-name
is the logs service user created in create step.
The output from the command may look like the following:
{
"createdAt": 1629913360323,
"updatedAt": 1629916772245,
"name": "controller-pattern",
"pluginName": "grok",
"rule": {
"pattern": "(?<ip>\\d{1,3}.\\d{1,3}.\\d{1,3}.\\d{1,3}) - - \\[(?<date>\\d{2}\\/\\w{3}\\/\\d{4}:\\d{2}:\\d{2}:\\d{2} \\+\\d{4})\\] \"(?<method>\\b\\w+\\b) \\/(?<service>\\b\\w+\\b?) (?<http>.*?)\" (?<code>\\d{3}) (?<bytes>\\d{3}) \"-\" \"(?<path>.*?)\" (?<count>\\d{3}) (?<message>.*)",
"target": "controller_meta"
},
"active": false
}