Opensearch - summary ingest

access.log 자동 summary

• filebeat로 access.log를 ingest 하는 동안 자동으로 summary까지 계산하게 한다.
• transform이나 job을 활용한다.
• pipeline을 활용한다.

transform

• access.log duration(평균응답속도)로 통계 산출(min/max/avg/count)
• start : POST _plugins/_transform/access-log-minute-summary/_start
• stop : POST _plugins/_transform/access-log-minute-summary/_stop
• status : GET _plugins/_transform -> enabled: true, continuous : true

DELETE _plugins/_transform/access-log-minute-summary
PUT _plugins/_transform/access-log-minute-summary
{
  "transform": {
    "enabled": true,
    "continuous": true,
    "schedule": {
      "interval": {
        "period": 1,
        "unit": "Minutes",
        "start_time": 1602100553
      }
    },
    "description": "Access log minute summary transform job",
    "source_index": "access-log-tomcat-*",
    "target_index": "access-log-sum-tomcat",
    "data_selection_query": {
      "match_all": {}
    },
    "page_size": 1000,
    "groups": [
      {
        "date_histogram": {
          "source_field": "@timestamp",
          "target_field": "target_timestamp",
          "fixed_interval": "5m",
          "timezone": "UTC"
        }
      }
    ],
    "aggregations": {
      "count": {
        "value_count": {
          "field": "status"
        }
      },
      "min_time": {
        "min": {
          "field": "my_duration"
        }
      },
      "max_time": {
        "max": {
          "field": "my_duration"
        }
      },
      "avg_time": {
        "avg": {
          "field": "my_duration"
        }
      }
    }
  }
}

pipeline

• millisecond time -> yyyy-MM-dd:HH:mm:ss 형태로 변경
• status : GET _ingest/pipeline
• del : DELETE _ingest/pipeline/convert_timestamp_pipeline

PUT _ingest/pipeline/convert_timestamp_pipeline
{
  "description": "Convert timestamp to human-readable format",
  "processors": [
    {
      "date": {
        "field": "target_timestamp",
        "target_field": "@timestamp",
        "formats": ["epoch_millis"],
        "output_format": "dd/MMM/yyyy:HH:mm:ss"
      }
    }
  ]
}

summary index에 pipeline 적용하기

PUT access-log-sum-tomcat/_settings
{
  "index.default_pipeline": "convert_timestamp_pipeline"
}

summary index 강제 생성(/with: pipeline)

DELETE access-log-sum-tomcat
GET /access-log-sum-tomcat
PUT /access-log-sum-tomcat
{
  "settings": {
    "index": {
      "number_of_shards": "1",
      "default_pipeline": "convert_timestamp_pipeline",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "target_timestamp": {
        "type": "long"
      },
      "avg_time": {
        "type": "float"
      },
      "count": {
        "type": "float"
      },
      "max_time": {
        "type": "float"
      },
      "min_time": {
        "type": "float"
      }
    }
  }
}