Search Logs

---

Overview

CloudAEye logs service provides programmatic access and searching capabilities using Query DSL for logs. This topic describes how user may query logs.

Prerequisites

User is required to create a logs service.

Search Logs

CloudAEye Dashboard

On Explore Logs page, you have few options to search and filter the logs.

1. Duration

You may select search query duration using a time period relative to now. If you click on the duration, you will see a drop-down menu. You may change the duration from the menu. If you would like to use absolute date and time, you may click on the calendar icon. That will show a calendar. You may choose the date from the calendar to use a specific date and time.

2. Search

You may search for any specific text that you are looking for. If there are any matches, the UI will show those log messages.

3. Tag Cloud

You may click on any tags from the tag cloud. This will filter the logs based on that tag. You will see the tag in the search box.

4. Filters

CloudAEye supports filtering the logs using a wide range of filters.

Here are the different types of filters you may use:

1. App Name: If you have more than one application, you may filter the logs based on application name.
2. Function Name: This allows filtering based on lambda function name.
3. Host: This allows filtering based on host name or IP address. For example, 192.168.19.32.
4. Index: You may filter the logs based on index name.
5. Level: You may choose log message levels - INFO, DEBUG, ERROR, etc.
6. Service Type: You may filter based on the type of AWS services. For example, lambda.
7. Anomaly Id: You may see the logs based on a particular anomaly id. This will show only the associated logs for an anomaly.
8. Request Id: You may see logs for a particular request id.
9. Region: If you have logs coming from multiple regions, you may filter based on specific AWS region(s).
10. AWS Service Group: You may filter logs based on a specific group you used for AWS Services logs.

You may apply one or more filters to narrow down your search namespace.

Query DSL

User may use JSON-based Query DSL to search logs.

Examples

Search for Documents

To perform a full text-search over logs data, use the below ElasticSearch REST API endpoint

GET {serviceEndpoint}/_search?q={search-term}

Example:

To search logs that have the term database in the logs data or meta data

GET https://demo-logs-service-km4s7fd5kydgbhfcug43nygbuq.us-east-2.es.amazonaws.com/_search?q=database

Search by Fields

To filter documents in a given index by specific field values, use the below ElasticSearch REST API

GET {serviceEndpoint}/{index}/_search

{
  "query": {
    "match": {
      "{field}": "{value}"
    }
  }
}

Example:

Search all the logs that have serviceName as api-gateway in index logstash-api

GET https://demo-logs-service-km4s7fd5kydgbhfcug43nygbuq.us-east-2.es.amazonaws.com/logstash-api/_search
{
  "query": {
    "match": {
      "serviceName": "api-gateway"
    }
  }
}

Paginating the Results

• To set the offset, use the from attribute and,
• To set the limit use the size attribute

To fetch limited / paginated documents in a given index by specific field values, use the below ElasticSearch REST API

GET {serviceEndpoint}/{index}/_search
{
  "from": {from},
  "size": {size},
  "query": {
    "match": {
      "{field}": "{value}"
    }
  }
}

Example:

Search all the logs that have serviceName as api-gateway and return only first 100 matching documents

GET https://demo-logs-service-km4s7fd5kydgbhfcug43nygbuq.us-east-2.es.amazonaws.com/logstash-*/_search
{
  "from": 0,
  "size": 100,
  "query": {
    "match": {
      "serviceName": "api-gateway"
    }
  }
}

Time Range Queries

One common use case while searching logs, is to filter logs between the given time range.

To filter documents between two given datetimes, use the below ElasticSearch REST API

GET {serviceEndpoint}/{index}/_search
{
    "query": {
        "range": {
            "{timeField}": {
                "gte": '{startDateTime}',
                "lt": '{endDateTime}'
            }
        }
    }
}

Example:

Search all the logs that streamed in for the last 24 hours (one day)

GET https://demo-logs-service-km4s7fd5kydgbhfcug43nygbuq.us-east-2.es.amazonaws.com/logstash-*/_search
{
    "query": {
        "range": {
            "time": {
                "gte": 'now-1d/d',
                "lt": 'now/d'
            }
        }
    }
}

ElasticSearch supports in-built pattern to search by different parts of a datetime value (like day, week, month etc). See the compete list of supported patterns here

One can also provide proper datetime values as string for the gte(greater than equal to) and lt (less than) fields

Search Documents by Ids

To fetch specific documents, by their unique ids, use the below ElasticSearch REST API

GET {serviceEndpoint}/{index}/_search
{
    "query": {
        "ids": {
            "values": []
        }
    }
}

Example:

Fetch multiple documents identified by the given doc-ids

GET https://demo-logs-service-km4s7fd5kydgbhfcug43nygbuq.us-east-2.es.amazonaws.com/logstash-*/_search
{
    "query": {
        "ids": {
            "values": [
                "Xasd23243fgq",
                "X9uted9955rh",
                "1ered45Dfdfg",
                "2glejgWt6vtu",
            ]
        }
    }
}