AWS CloudWatch 경보 Slack 알림 설정

OpenSearch를 사용한 ElasticSearch 사용 시 설정한 경보에 따른 알림 설정 프로세스 작성

Amazon OpenSearch Service에 직접 들어가지 않으면 실행중인 엔진의 상태를 모니터링 할 수 없습니다.
스토리지 등 핵심 성능 지표를 CloudWatch 경보로 설정하고 설정된 경보가 발생하면 Slack으로 메시지를 보내는 프로세스를 작성합니다.

1. Amazon SNS 주제 생성
   

2. CloudWatch 지표 생성
   
   2-1. 지표 선택 (opensearchservice freestorage)
   
   2-2. 조건 설정
   
   2-3. 알림 설정
   
   2-4. 이름 설정 및 경보 생성
   

3. KMS 생성
   

4. Lambda 생성
   
   4-1. 기본 정보 및 SNS 트리거 설정
   
   4-2. 환경 변수 설정
   

5. Lambda 함수 설정

'''
Follow these steps to configure the webhook in Slack:

  1. Navigate to https://<your-team-domain>.slack.com/services/new

  2. Search for and select "Incoming WebHooks".

  3. Choose the default channel where messages will be sent and click "Add Incoming WebHooks Integration".

  4. Copy the webhook URL from the setup instructions and use it in the next section.

To encrypt your secrets use the following steps:

  1. Create or use an existing KMS Key - http://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html

  2. Expand "Encryption configuration" and click the "Enable helpers for encryption in transit" checkbox

  3. Paste <SLACK_CHANNEL> into the slackChannel environment variable

  Note: The Slack channel does not contain private info, so do NOT click encrypt

  4. Paste <SLACK_HOOK_URL> into the kmsEncryptedHookUrl environment variable and click "Encrypt"

  Note: You must exclude the protocol from the URL (e.g. "hooks.slack.com/services/abc123").

  5. Give your function's role permission for the `kms:Decrypt` action using the provided policy template
'''

import boto3
import json
import logging
import os

from base64 import b64decode
from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError


# The base-64 encoded, encrypted key (CiphertextBlob) stored in the kmsEncryptedHookUrl environment variable
ENCRYPTED_HOOK_URL = os.environ['kmsEncryptedHookUrl']
# The Slack channel to send a message to stored in the slackChannel environment variable
SLACK_CHANNEL = os.environ['slackChannel']

HOOK_URL = "https://" + boto3.client('kms').decrypt(
    CiphertextBlob=b64decode(ENCRYPTED_HOOK_URL),
    EncryptionContext={'LambdaFunctionName': os.environ['AWS_LAMBDA_FUNCTION_NAME']}
)['Plaintext'].decode('utf-8')


logger = logging.getLogger()
logger.setLevel(logging.INFO)


def lambda_handler(event, context):
    logger.info("Event: " + str(event))
    message = json.loads(event['Records'][0]['Sns']['Message'])
    logger.info("Message: " + str(message))

    alarm_name = message['AlarmName']
    #old_state = message['OldStateValue']
    new_state = message['NewStateValue']
    reason = message['NewStateReason']

    slack_message = {
        'channel': SLACK_CHANNEL,
		"attachments": [
			{
				"color": "FF0000",
				"blocks": [
					{
						"type": "section",
						"text": {
							"type": "plain_text",
							"text": reason
						}
					},
					{
						"type": "section",
						"text": {
							"type": "mrkdwn",
							"text": "*CloudWatch:* " + alarm_name
						}
					},
					{
						"type": "section",
						"fields": [
							{
								"type": "mrkdwn",
								"text": "*Status:* " + new_state
							},
							{
								"type": "mrkdwn",
								"text": "*Link:* <https://ap-northeast-2.console.aws.amazon.com/cloudwatch/home?region=ap-northeast-2#alarmsV2:alarm/" + alarm_name + "| AWS CloudWatch>"
							}
						]
					},
					{
						"type": "context",
						"elements": [
							{
								"type": "plain_text",
								"text": "ES FreeStorageSpace warning"
							}
						]
					}
				]
			}
		]
    }

    req = Request(HOOK_URL, json.dumps(slack_message).encode('utf-8'))
    try:
        response = urlopen(req)
        response.read()
        logger.info("Message posted to %s", slack_message['channel'])
    except HTTPError as e:
        logger.error("Request failed: %d %s", e.code, e.reason)
    except URLError as e:
        logger.error("Server connection failed: %s", e.reason)

6. 결과 확인 및 테스트
   

7. OpenSearch 외 다른 항목도 CloudWatch 등록 후 경보 설정 및 알림 설정 가능

   • 목적별 SNS, Lambda 생성