상황

Admin/develop Build 실패

BUILD SUCCESSFUL in 40s
3 actionable tasks: 3 executed
+ docker build -t accountID.dkr.ecr.ap-northeast-2.amazonaws.com/xx-admin:c1b51a3 .
Sending build context to Docker daemon  893.1MB


Step 1/5 : FROM adoptopenjdk:11-jre-hotspot
 ---> 3be5117ed0ca

Step 2/5 : RUN ln -snf /usr/share/zoneinfo/Asia/Seoul /etc/localtime &&     echo "Asia/Seoul" > /etc/timezone
 ---> Using cache
 ---> fc09c9a74d04

Step 3/5 : RUN mkdir -p /xxxx/crawl
 ---> Using cache
 ---> fc7088136587

Step 4/5 : COPY build/resources/main/static/editor* /xxxx/
 ---> Using cache
 ---> 562111c821cc

Step 5/5 : COPY build/libs/Admin.jar Admin.jar
Error processing tar file(exit status 1): write /Admin.jar: no space left on device

script returned exit code 1

원인

Jenkins 서버 루트볼륨 용량 부족으로 한 Script 종료 코드 반환

• 서버점검 결과 : 운영 Jenkins 서버 볼륨 부족 현상으로 인해 G2Admin 이외 전체 파이프라인 빌드 실패

디스크 공간 사용량 99% 원인

• Jenkins 서버 내 docker image 생성 시 빌드캐시가 존재하여 디스크 공간이 부족하였음.

[ec2-user@ip-172-16-11-239 ~]$ docker image ls
REPOSITORY                                                     TAG                 IMAGE ID            CREATED             SIZE
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-admin     b823e77             ff3bbc0afc91        3 hours ago         468MB
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-admin     602ce59             2b169e6eb01f        3 hours ago         468MB
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-admin     c7a2ce1             9e2a4addae46        6 days ago          468MB
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-admin     4c68251             5df1719e6992        6 days ago          491MB
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-admin     fba29c0             74248f796982        6 days ago          468MB
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-admin     e507745             b041aa207ee4        7 days ago          468MB
... 중략
<none>                                                      <none>              543f9062aa7e        6 weeks ago         275MB
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-mq        fdcc072             81ff6795ff8e        6 weeks ago         275MB
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-admin     82ef944             6872b10e86a8        2 months ago        465MB
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-api       6bcc84b             a257156f7524        3 months ago        276MB
<none>                                                      <none>              ddcf848309fd        3 months ago        276MB
<none>                                                      <none>              88be7ee42081        3 months ago        276MB
<none>                                                      <none>              ef844226e53b        3 months ago        276MB
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-admin     <none>              449d142b835b        3 months ago        433MB
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-admin     <none>              38749ca2a84e        3 months ago        433MB
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-admin     <none>              3b8593db7a1e        3 months ago        433MB
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-admin     <none>              b9d5605404e5        3 months ago        433MB
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-admin     <none>              76eff367f657        3 months ago        433MB
<none>                                                      <none>              79acd8435fb5        5 months ago        461MB
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-admin     <none>              c54817b2df1d        5 months ago        432MB
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-auth      4329125             727438fdb640        5 months ago        281MB
jenkins/jenkins                                             jdk11               230a05eb9462        6 months ago        685MB
adoptopenjdk                                                latest              a684d5649a9a        6 months ago        474MB
jenkins/jenkins                                             <none>              7ad786757af5        6 months ago        685MB
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-admin     7ee1f5b             a5337486c652        7 months ago        442MB
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-stats     1023e9e             6abe1d8e9b25        8 months ago        265MB
accountID.dkr.ecr.ap-northeast-2.amazonaws.com/g2-service   3fa7907             b545b6f66edc        8 months ago        271MB
<none>                                                      <none>              7a6d0c8fb40a        14 months ago       737MB
adoptopenjdk                                                11-jre-hotspot      3be5117ed0ca        15 months ago       225MB

해결방법

Docker Build Cache 정리

• 매월 말일 23:55분 Jenkins Docker Build Cache Cron 작업 설정 등록

55 23 28-31 * * [[ $(date +'%d') -eq $(cal | awk '!/^$/{ print $NF }' | sed '/^[[:space:]]*$/d' | tail -1) ]] && /usr/bin/docker system prune -a -f

Jenkins Disk 볼륨 확인

$ df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
devtmpfs       devtmpfs  1.9G     0  1.9G   0% /dev
tmpfs          tmpfs     1.9G     0  1.9G   0% /dev/shm
tmpfs          tmpfs     1.9G  476K  1.9G   1% /run
tmpfs          tmpfs     1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/nvme0n1p1 xfs       100G   65G   36G  65% /
tmpfs          tmpfs     389M     0  389M   0% /run/user/1000

Docker Cache 사용량 확인

$ du -sh * | sort -rh | head -n 10
43G     overlay2
58M     image
53M     containers
884K    volumes
56K     network
56K     buildkit
16K     builder
0       trust
0       tmp
0       swarm

overlay2 Cache 사용량 정리

$ docker system prune -a -f
... 중략
deleted: sha256:57e12361aa14d1b57125594b11d71035c8b87e88c1f72742315e9b29a2b8a6ba
deleted: sha256:d3991ad41f89923dac46b632e2b9869067e94fcdffa3ef56cd2d35b26dd9bce7
deleted: sha256:2e533c5c9cc8936671e2012d79fc6ec6a3c8ed432aa81164289056c71ed5f539
deleted: sha256:282c79e973cf51d330b99d2a90e6d25863388f66b1433ae5163ded929ea7e64b
deleted: sha256:cc4590d6a7187ce8879dd8ea931ffaa18bc52a1c1df702c9d538b2f0c927709d

Total reclaimed space: 43.65GB

결과

• /var/lib/docker/overlay2 경로 상 Jenkins 빌드 캐시 쌓인 부분이 삭제되면서 총 43.65GB 디스크 공간을 확보하였음.

Jenkins Build 확인

• Jenkins 볼륨 확장 이후 전체 파이프라인 빌드 성공