Skip to content

caeops > logs

create-parsing-rule

Description

This command creates a parsing rule filter for the given logs source

Synopsis

  create-parsing-rule
--name [value]
--service-name [value]
--plugin-name [value]
--filter [value]

Options

--name (string)

Name of parsing rule

--service-name (string)

Name of the logs source to create this rule for

--plugin-name (string)

Name of the plugin (logstash plugin) to use for parsing log message. Supported values : grok

--filter (string)

The format of parsing to be applied by the rule to extract relevant information from the log message

Examples

The following logs create-parsing-rule example creates a parsing rule

caeops logs create-parsing-rule 
    --name access-pattern --service-name dev-logs --plugin-name grok 
    --filter '{"pattern": "(?<date>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3})  (?<method>\b\w+\b) \[(?<service>.*?)\] (?<count>[0-9]+) --- \[(?<hook>.*?)\] (?<action>.*) : (?<log_message>.*)","target": "parsed_data"}'

--filter :

  • The pattern can take any inbuilt or custom pattern supported by the logstash's Grok filter plugin. See the complete list of inbuilt patterns.

  • The target determines the meta-data field name that should contain the parsed log object

In the above example for the given pattern, if the log message looks like below 

2021-08-10 12:08:50.384  INFO [api-gateway,,,] 1 --- [extShutdownHook] o.s.b.w.embedded.netty.GracefulShutdown  : Commencing graceful shutdown. Waiting for active requests to complete

then the parsed log message structure would be 

{
    'date': '2021-08-10 12:08:50.384',
    'method': 'INFO',
    'service': 'api-gateway,,,'
    'count': 1
    'hook': 'extShutdownHook'
    'action': 'o.s.b.w.embedded.netty.GracefulShutdown',
    'log_message': 'Commencing graceful shutdown. Waiting for active requests to complete'
}

Logstash's Grok filter uses Oniguruma regex library. In case you want to build custom regex like in the above example, use the rubular to create and validate your regex patterns.

Output

Parsing Rule Details -> (Structure)

  • name -> (string) Name of the parsing rule created
  • serviceName -> (string) Name of the logs source that applies this rule
  • pluginName -> (string) Name of the plugin(logstash) used for parsing logs
  • rule -> (structure)
    • pattern -> (string) Filter pattern defined for this rule
    • target -> (string) Name of the target field, that should contain the parsed information
Back to top