Skip to content

Search Logs


Overview

CloudAEye logs service provides programmatic access and searching capabilities using Query DSL for logs. This topic describes how user may query logs.

⚓ Prerequisites

User is required to create a logs service.

Search Logs

Query DSL

User may use JSON-based Query DSL from Open Distro for Elasticsearch (see official docs from Elastic) to search logs.

Examples

Search for Documents

To perform a full text-search over logs data, use the below ElasticSearch REST API endpoint

GET {serviceEndpoint}/_search?q={search-term}

Example:

To search logs that have the term database in the logs data or meta data

GET https://demo-logs-service-km4s7fd5kydgbhfcug43nygbuq.us-east-2.es.amazonaws.com/_search?q=database

Search by Fields

To filter documents in a given index by specific field values, use the below ElasticSearch REST API

GET {serviceEndpoint}/{index}/_search

{
  "query": {
    "match": {
      "{field}": "{value}"
    }
  }
}

Example:

Search all the logs that have serviceName as api-gateway in index logstash-api

GET https://demo-logs-service-km4s7fd5kydgbhfcug43nygbuq.us-east-2.es.amazonaws.com/logstash-api/_search
{
  "query": {
    "match": {
      "serviceName": "api-gateway"
    }
  }
}

Paginating the Results

  • To set the offset, use the from attribute and,
  • To set the limit use the size attribute

To fetch limited / paginated documents in a given index by specific field values, use the below ElasticSearch REST API

GET {serviceEndpoint}/{index}/_search
{
  "from": {from},
  "size": {size},
  "query": {
    "match": {
      "{field}": "{value}"
    }
  }
}

Example:

Search all the logs that have serviceName as api-gateway and return only first 100 matching documents

GET https://demo-logs-service-km4s7fd5kydgbhfcug43nygbuq.us-east-2.es.amazonaws.com/logstash-*/_search
{
  "from": 0,
  "size": 100,
  "query": {
    "match": {
      "serviceName": "api-gateway"
    }
  }
}

Time Range Queries

One common use case while searching logs, is to filter logs between the given time range.

To filter documents between two given datetimes, use the below ElasticSearch REST API

GET {serviceEndpoint}/{index}/_search
{
    "query": {
        "range": {
            "{timeField}": {
                "gte": '{startDateTime}',
                "lt": '{endDateTime}'
            }
        }
    }
}
Example:

Search all the logs that streamed in for the last 24 hours (one day)

GET https://demo-logs-service-km4s7fd5kydgbhfcug43nygbuq.us-east-2.es.amazonaws.com/logstash-*/_search
{
    "query": {
        "range": {
            "time": {
                "gte": 'now-1d/d',
                "lt": 'now/d'
            }
        }
    }
}

ElasticSearch supports in-built pattern to search by different parts of a datetime value (like day, week, month etc). See the compete list of supported patterns here

One can also provide proper datetime values as string for the gte(greater than equal to) and lt (less than) fields

Search Documents by Ids

To fetch specific documents, by their unique ids, use the below ElasticSearch REST API

GET {serviceEndpoint}/{index}/_search
{
    "query": {
        "ids": {
            "values": []
        }
    }
}
Example:

Fetch multiple documents identified by the given doc-ids

GET https://demo-logs-service-km4s7fd5kydgbhfcug43nygbuq.us-east-2.es.amazonaws.com/logstash-*/_search
{
    "query": {
        "ids": {
            "values": [
                "Xasd23243fgq",
                "X9uted9955rh",
                "1ered45Dfdfg",
                "2glejgWt6vtu",
            ]
        }
    }
}