(opening storage failed)
ts=2022-10-06T03:24:48.479Z caller=main.go:799 level=info msg="Stopping scrape discovery manager..."
ts=2022-10-06T03:24:48.479Z caller=main.go:813 level=info msg="Stopping notify discovery manager..."
ts=2022-10-06T03:24:48.479Z caller=main.go:835 level=info msg="Stopping scrape manager..."
ts=2022-10-06T03:24:48.479Z caller=main.go:809 level=info msg="Notify discovery manager stopped"
ts=2022-10-06T03:24:48.479Z caller=main.go:795 level=info msg="Scrape discovery manager stopped"
ts=2022-10-06T03:24:48.479Z caller=main.go:829 level=info msg="Scrape manager stopped"
ts=2022-10-06T03:24:48.479Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..."
ts=2022-10-06T03:24:48.479Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped"
ts=2022-10-06T03:24:48.479Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..."
ts=2022-10-06T03:24:48.479Z caller=main.go:1055 level=info msg="Notifier manager stopped"
ts=2022-10-06T03:24:48.479Z caller=main.go:1064 level=error err="opening storage failed: found unsequential head chunk files /data/prometheus/prometheus/chunks_head/000009 (index: 9) and /data/prometheus/prometheus/chunks_head/000019 (index:다음같은 에러로그가 찍히며 prometheus 실행이 안될 경우
에러문에 찍힌 말처럼 "opening storage failed: found unsequential head chunk files /data/prometheus/prometheus/chunks_head/000009 (index: 9) "
./prometheus/data 디렉토리 안에 lock 파일이 동시성 문제가 발생하지 않게 하기위한 파일인데 비정상 종료로 인해 삭제되지 않고 남아있게됨으로써 발생하는 문제입니다.
이 경우 prometheus를 내린다음 ./prometheus/data 디렉토리 안에 lock파일을 삭제 후 prometheus를 재실행 하면 됩니다.
DevOps 🐥