가시다(gasida) 님이 진행하는 Terraform T101 4기 실습 스터디 게시글입니다.
책 '테라폼으로 시작하는 IaC' 를 참고하였고, 스터디하면서 도움이 될 내용들을 정리하고자 합니다.
5주차는 Terraform Module과 & Runner에 대해 학습을 하였습니다.
앞전 주차 코드들이 Module 기반으로 작성되어 있어서, 5주차는 Terraform Runner인 Atlantis에 대해 심화 학습하면서 내용을 정리하겠습니다.
코드 작성시 고려한 사항
- 실습환경 : Github + AWS + Atlantis + Terraform
- Altantis 위한 EC2 환경을 CloudFormation 대신 Terraform 코드로 작성
- VPC 모듈 이용하여 전용 CICD 환경 구성
- 보안성 고려하여 Access Key 생성 되신 EC2 Instance에 Role 부여
- Atlantis 이용하여 개발 EKS Cluster 배포 테스트
<출처> https://www.runatlantis.io/blog/2017/introducing-atlantis.html
Atlantis는 Hootsuite에서 1년 이상 사용되어 온 Terraform 에서 협업하기 위한 도구입니다. Atlantis의 핵심 기능을 통해 개발자와 운영자는 Terraform Pull Requests(풀 리퀘스트)에서 직접 terraform plan과 apply를 할 수 있고, 그런 다음 Atlantis는 명령의 출력으로 풀 리퀘스트에 다시 주석을 달아 줍니다.
Terraform 워크플로를 풀 리퀘스트에 도입함으로써 Atlantis는 운영팀이 Terraform에서 더 잘 협업할 수 있도록 도왔고, 전체 개발팀이 Terraform을 안전하게 작성하고 실행할 수 있도록 했습니다.
Atlantis는 Hootsuite에서 Terraform을 도입하면서 발생한 두 가지 문제를 해결하기 위해 구축되었습니다.
효과적인 협업
Terraform에서 팀으로 협업하는 가장 좋은 방법은 무엇입니까?
Terraform을 작성하는 개발자
개발자들이 Terraform을 안전하게 작성하고 적용할 수 있도록 어떻게 지원할 수 있을까요?
Terraform을 작성할 때 따를 수 있는 워크플로가 여러 개 있습니다. 가장 간단한 워크플로는 master를 사용하는 것입니다.
이 워크플로에서는 master에서 작업하고 로컬에서 terraform을 실행하는 것입니다. 이 워크플로의 문제점은 협업이나 코드 검토가 없다는 것입니다. 그래서 풀 리퀘스트를 사용하기 시작합니다.
우리는 여전히 로컬에서 terraform plan을 실행 하지만, 변경 사항에 만족하면서 검토를 위한 풀 리퀘스트를 만듭니다. 풀 리퀘스트가 승인되면 로컬에서 apply를 실행합니다.
이 워크플로는 개선되었지만 여전히 문제가 있습니다. 첫 번째 문제는 풀 리퀘스트에서 diff 만 검토하기 어렵다는 것입니다. 변경 사항을 제대로 검토하려면 실제로 terraform plan의 출력 결과를 봐야 합니다.
작은 변화처럼 보이는데...
...큰 계획(Change)를 유발할 수 있습니다.
두 번째 문제는 이제 master가 실제로 적용된 것과 동기화되지 않는 것이 쉽다는 것입니다. apply를 실행하지 않고 풀 요청을 병합하거나 apply에 오류가 있는 경우 수정하는 것을 잊어버린 후 master에 병합하는 경우 이런 일이 발생할 수 있습니다. 이제 master에 있는 것은 실제로 프로덕션에서 실행되는 것이 아닙니다. 기껏해야 다음에 누군가가 Terraform plan을 실행할 때 혼란을 야기합니다. 최악의 경우 누군가가 master에 있는 것이 실제로 실행되고 있다고 가정하고 이에 의존할 때 중단이 발생합니다.
Atlantis 워크플로를 사용하면 다음과 같은 문제가 해결됩니다.
이제 풀 리퀘스트에 대한 terraform plan의 출력을 볼 수 있으므로 변경 사항을 검토하기가 쉽습니다.
풀 리퀘스트는 계획을 볼 수 있으므로 검토하기 쉽습니다.
또한 풀 요청에서 실제 apply 출력을 볼 수 있으므로 마스터에 병합하기 전에 풀 요청이 테라폼 적용되었는지 쉽게 확인할 수 있습니다.
그렇다면 Atlantis는 운영팀 내에서 Terraform 작업을 훨씬 더 쉽게 만들어 주지만, 전체 팀이 Terraform을 작성하도록 하는 데는 어떤 도움이 될까요?
Terraform은 일반적으로 Ops팀에서 사용되기 시작합니다. Terraform을 사용한 결과 Ops팀은 인프라 변경 작업 속도가 훨씬 빨라졌지만 개발자가 이러한 변경을 요청하는 방식은 동일하게 유지되었습니다. 즉, 티켓팅 시스템이나 채팅을 사용하여 운영팀에 도움을 요청하고 요청은 대기열로 이동한 후 나중에 진행됩니다. Ops는 작업이 완료되었다고 응답합니다.
그러나 곧 Ops팀은 개발자가 이러한 Terraform 변경 사항 중 일부를 직접 수행하는 것이 가능하다는 사실을 깨닫기 시작했습니다! 하지만 발생하는 몇 가지 문제가 있습니다.
개발자에게는 실제로 Terraform 명령을 실행할 수 있는 자격 증명이 없습니다.
자격 증명을 제공하면 실제로 적용되는 내용을 검토하기가 어렵습니다.
Atlantis를 사용하면 이러한 문제가 해결됩니다. 모든 terraform plan 및 apply 명령은 풀 요청에서 실행됩니다. 즉, 개발자는 Terraform을 로컬에서 실행하기 위해 자격 증명이 필요하지 않습니다. 물론 이는 위험할 수 있습니다. 개발자(Terraform을 처음 접하는 사람)가 적용하지 말아야 할 사항을 적용하지 않도록 어떻게 보장할 수 있습니까? 대답은 코드 검토 및 승인입니다.
Atlantis는 풀 리퀘스트에 대한 plan에 대해 직접 응답하므로 운영 엔지니어가 어떤 변경 사항이 적용될지 정확하게 검토하기가 쉽습니다. 그리고 Atlantis는 require-approval 모드에서 실행될 수 있으며, apply 을 실행하기 전에 GitHub 풀 리퀘스트 승인이 필요합니다.
Atlantis를 사용하면 개발자는 Terraform을 안전하게 작성하고 적용할 수 있습니다. 풀 리퀘스트를 제출하고 변경 사항이 좋아 보일 때까지 atlantis plan을 실행한 다음 Ops의 승인을 받아 apply할 수 있습니다.
Security Group Inbound Rule로 4141/tcp 추가
EC2는 Github에서 접속가능하도록 EIP 필요
보안 고려하여 Terraform 권한위해 credential 대신 iam role 활용
#!/bin/bash
hostnamectl --static set-hostname Atlantis
# Config convenience
echo 'alias vi=vim' >> /etc/profile
echo "sudo su -" >> /home/ubuntu/.bashrc
# Install Packages & Terraform
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
apt update -qq && apt install tree jq unzip zip terraform -y
# Install aws cli version2
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
rm awscliv2.zip
sudo ./aws/install
# Install atlantis
wget https://github.com/runatlantis/atlantis/releases/download/v0.28.3/atlantis_linux_amd64.zip -P /root
unzip /root/atlantis_linux_amd64.zip -d /root && rm -rf /root/atlantis_linux_amd64.zip
URL="http://$(curl -s ipinfo.io/ip):4141"
USERNAME=icebreaker70
TOKEN='ghp_55DZZ***********TzVm5X0R1oSa5g'
SECRET='nsth************nnfbqazndauff'
REPO_ALLOWLIST="github.com/icebreaker70/t101-cicd"
echo $URL $USERNAME $TOKEN $SECRET $REPO_ALLOWLIST
nohup /root/atlantis server --atlantis-url="$URL" --gh-user="$USERNAME" --gh-token="$TOKEN" --gh-webhook-secret="$SECRET" --repo-allowlist="$REPO_ALLOWLIST" &
erraform {
required_version = ">= 1.8"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.34"
}
}
}
provider "aws" {
region = var.region
default_tags {
tags = {
Environment = var.env
Project = var.pjt
Service = var.svc
TerraformManaged = true
}
}
}
################################################################################
# Common data/locals
################################################################################
data "aws_availability_zones" "available" {}
locals {
name = "t101-sjkim-${basename(path.cwd)}"
region = var.region
vpc_cidr = "10.0.0.0/16"
azs = slice(data.aws_availability_zones.available.names, 0, 2)
tags = {
Environment = var.env
Project = var.pjt
ServiceProvider = var.svc
TerraformManaged = true
}
}
variable "profile" {
description = "The name of the AWS profile in the credentials file"
type = string
default = "t101"
}
variable "region" {
description = "The name of the AWS Default Region"
type = string
default = "ap-northeast-2"
}
# default tag
variable "env" {
default = "common"
}
variable "pjt" {
default = "t101"
}
variable "svc" {
default = "cicd"
}
variable "tags" {
description = "Default tags attached to all resources."
type = map(string)
default = {
Environment = "common",
Project = "t101",
Service = "cicd",
TerraformManaged = "true"
}
}
variable "my_ip" {
description = "my public ip"
type = string
default = "1.1.1.1/32"
}
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "~> 5.0"
name = local.name
cidr = local.vpc_cidr
azs = local.azs
private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)]
public_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 48)]
enable_nat_gateway = false
single_nat_gateway = true
map_public_ip_on_launch = true
tags = local.tags
}
Ubuntu 22.04 최신버전 가져오기
data "aws_ami" "ubuntu" {
most_recent = true
owners = ["099720109477"] // AMI 소유자
filter {
name = "name" // AMI 이름
values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-*"]
}
filter {
name = "virtualization-type" // 가상화 타입
values = ["hvm"]
}
filter {
name = "architecture" // 이미지 아키텍처
values = ["x86_64"]
}
}
resource "aws_instance" "atlantis" {
ami = data.aws_ami.ubuntu.id
instance_type = "t3.micro"
vpc_security_group_ids = [aws_security_group.ec2_atlantis.id]
subnet_id = module.vpc.public_subnets[0]
key_name = "martha"
iam_instance_profile = aws_iam_instance_profile.profile_ec2_terraform.name
associate_public_ip_address = true
enable_volume_tags = true
user_data = <<-EOT
#!/bin/bash
hostnamectl --static set-hostname Atlantis
# Config convenience
echo 'alias vi=vim' >> /etc/profile
echo "sudo su -" >> /home/ubuntu/.bashrc
# Install Packages & Terraform
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
apt update -qq && apt install tree jq unzip zip terraform -y
# Install aws cli version2
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
# Install atlantis
wget https://github.com/runatlantis/atlantis/releases/download/v0.28.3/atlantis_linux_amd64.zip -P /root
unzip /root/atlantis_linux_amd64.zip -d /root && rm -rf /root/atlantis_linux_amd64.zip
URL="http://$(curl -s ipinfo.io/ip):4141"
USERNAME=icebreaker70
TOKEN='ghp_55DZZ***********TzVm5X0R1oSa5g'
SECRET='nsth************nnfbqazndauff'
REPO_ALLOWLIST="github.com/icebreaker70/t101-cicd"
echo $URL $USERNAME $TOKEN $SECRET $REPO_ALLOWLIST
nohup /root/atlantis server --atlantis-url="$URL" --gh-user="$USERNAME" --gh-token="$TOKEN" --gh-webhook-secret="$SECRET" --repo-allowlist="$REPO_ALLOWLIST" &
EOT
tags = {
Name = "ec2-${var.env}-${var.pjt}-${var.svc}-atlantis"
}
}
resource "aws_security_group" "ec2_atlantis" {
name = "SG-ec2-${var.env}-${var.pjt}-${var.svc}-atlantis"
vpc_id = module.vpc.vpc_id
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = [var.my_ip]
}
ingress {
from_port = 4141
to_port = 4141
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
protocol = "-1"
from_port = 0
to_port = 0
cidr_blocks = ["0.0.0.0/0"]
}
}
variable "security_group_name" {
description = "The name of the security group"
type = string
default = "SG-dev-t101-sjkim-bastion"
}
# IAM Role 생성
resource "aws_iam_role" "role_ec2_terraform" {
name = "role_ec2_terraform"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow",
Principal = {
Service = "ec2.amazonaws.com" # IAM Role 생성 시, EC2 Profile 역할로 사용됨을 지정
}
Action = "sts:AssumeRole" # AdministratorAccess 권한이 존재 하더라도, AssumeRole로 접근 필수적
}
]
})
tags = {
Name = "role-ec2-atlantis" # 생성한 IAM에 지정되는 Tag 이름
}
}
# 위에서 생성한 IAM Role에 필요 정책 부착
resource "aws_iam_role_policy_attachment" "attach-administrator-policy" {
policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess"
role = aws_iam_role.role_ec2_terraform.name
}
resource "aws_iam_instance_profile" "profile_ec2_terraform" {
name = "profile_ec2_terraform"
role = aws_iam_role.role_ec2_terraform.name
}
output "public_ip" {
value = aws_instance.atlantis.public_ip
description = "The public IP of the Instance"
}
$ terraform init && terraform validate && terraform fmt
$ terraform plan -out tfplan
$ terraform apply tfplan
data.aws_ami.ubuntu
data.aws_availability_zones.available
aws_iam_instance_profile.profile_ec2_terraform
aws_iam_role.role_ec2_terraform
aws_iam_role_policy_attachment.attach-administrator-policy
aws_instance.atlantis
aws_security_group.ec2_atlantis
module.vpc.aws_default_network_acl.this[0]
module.vpc.aws_default_route_table.default[0]
module.vpc.aws_default_security_group.this[0]
module.vpc.aws_internet_gateway.this[0]
module.vpc.aws_route.public_internet_gateway[0]
module.vpc.aws_route_table.private[0]
module.vpc.aws_route_table.public[0]
module.vpc.aws_route_table_association.private[0]
module.vpc.aws_route_table_association.private[1]
module.vpc.aws_route_table_association.public[0]
module.vpc.aws_route_table_association.public[1]
module.vpc.aws_subnet.private[0]
module.vpc.aws_subnet.private[1]
module.vpc.aws_subnet.public[0]
module.vpc.aws_subnet.public[1]
module.vpc.aws_vpc.this[0]
vpc
EC2 for Atlantis
Security Group
Instance Profile
EC2 접속
$ ssh -i ~/keypair/martha.pem ubuntu@$(terraform output -raw public_ip)
Welcome to Ubuntu 22.04.4 LTS (GNU/Linux 6.5.0-1022-aws x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/pro
System information as of Sat Jul 13 10:44:34 UTC 2024
System load: 0.01 Processes: 111
Usage of /: 32.8% of 7.57GB Users logged in: 1
Memory usage: 26% IPv4 address for ens5: 10.0.48.83
Swap usage: 0%
Expanded Security Maintenance for Applications is not enabled.
0 updates can be applied immediately.
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
Last login: Sat Jul 13 07:07:11 2024 from 1.1.1.1
root@Atlantis:~#
- Go to your repo's settings
- Select Webhooks or Hooks in the sidebar
- Click Add webhook
- set Payload URL to your ngrok url with
/events
at the end. Ex.http://**<EC2공인IP>:4141**/events
- double-check you added
/events
to the end of your URL.- set Content type to
application/json
- set Secret to your random string
- select Let me select individual events
- check the boxes
- Issue comments
- Pull request reviews
- Pushes
- Pull requests
- leave Active checked
- click Add webhook
USERNAME="{the username of your GitHub, GitLab or Bitbucket user}"
REPO_ALLOWLIST="$YOUR_GIT_HOST/$YOUR_USERNAME/$YOUR_REPO"
***REPO_ALLOWLIST="github.com/icebreaker70/t101-cicd"***
# ex. REPO_ALLOWLIST="github.com/runatlantis/atlantis"
# If you're using Bitbucket Server, $YOUR_GIT_HOST will be the domain name of your
# server without scheme or port and $YOUR_USERNAME will be the name of the **project** the repo
# is under, **not the key** of the project.
root@Atlantis:~# USERNAME=icebreaker70
root@Atlantis:~# REPO_ALLOWLIST="github.com/icebreaker70/t101-cicd"
root@Atlantis:~# URL="http://$(curl -s ipinfo.io/ip):4141"
TOKEN='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'
root@Atlantis:~# SECRET='nstheyfiwssekvtfpvnnnfbqazndauff'
root@Atlantis:~# echo $URL $USERNAME $TOKEN $SECRET $REPO_ALLOWLIST
http://13.209.9.46:4141 icebreaker70 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 nstheyfiwssekvtfpvnnnfbqazndauff github.com/icebreaker70/t101-cicd
root@Atlantis:~# ./atlantis server \
--atlantis-url="$URL" \
--gh-user="$USERNAME" \
--gh-token="$TOKEN" \
--gh-webhook-secret="$SECRET" \
--repo-allowlist="$REPO_ALLOWLIST"
{"level":"info","ts":"2024-07-13T07:01:11.924Z","caller":"server/server.go:319","msg":"Supported VCS Hosts%!(EXTRA string=hosts, []models.VCSHostType=[Github])","json":{}}
{"level":"info","ts":"2024-07-13T07:01:12.382Z","caller":"server/server.go:467","msg":"Utilizing BoltDB","json":{}}
{"level":"info","ts":"2024-07-13T07:01:12.392Z","caller":"policy/conftest_client.go:153","msg":"failed to get default conftest version. Will attempt request scoped lazy loads DEFAULT_CONFTEST_VERSION not set","json":{}}
{"level":"info","ts":"2024-07-13T07:01:12.393Z","caller":"server/server.go:1017","msg":"Atlantis started - listening on port 4141","json":{}}
{"level":"info","ts":"2024-07-13T07:01:12.393Z","caller":"scheduled/executor_service.go:51","msg":"Scheduled Executor Service started","json":{}}
root@Atlantis:~# ss -tnlp
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=319,fd=14))
LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=609,fd=3))
LISTEN 0 4096 *:4141 *:* users:(("atlantis",pid=1939,fd=7))
LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=609,fd=4))
root@Atlantis:~# # 웹 접속 확인
URL="http://$(curl -s ipinfo.io/ip):4141"
echo $URL
http://13.209.9.46:4141
Atlantis WebPage
Github Webhook ping test
git clone
$ git clone https://github.com/icebreaker70/t101-cicd && cd t101-cicd && treefeature branch 생성
$ git branch test && git checkout test && git branchmain.tf 파일 작성
$ echo 'resource "null_resource" "example" {}' > main.tfadd commit push
$ git add main.tf && git commit -m "add main.tf" && git push origin test
$ git clone https://github.com/icebreaker70/t101-cicd && cd t101-cicd && tree
Cloning into 't101-cicd'...
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 4 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (4/4), done.
.
└── README.md
1 directory, 1 file
$ git branch test && git checkout test && git branch
Switched to branch 'test'
main
* test
$ echo 'resource "null_resource" "example" {}' > main.tf
$ git add main.tf && git commit -m "add main.tf" && git push origin test
[test ff7416c] add main.tf
1 file changed, 1 insertion(+)
create mode 100644 main.tf
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 8 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 346 bytes | 346.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote:
remote: Create a pull request for 'test' on GitHub by visiting:
remote: https://github.com/icebreaker70/t101-cicd/pull/new/test
remote:
To https://github.com/icebreaker70/t101-cicd
* [new branch] test -> test
$ watch -d tree .atlantis/
.atlantis/
├── atlantis.db
├── bin
└── plugin-cache
2 directories, 1 file
Compare & pull request 클릭
Create pull request : title ( create null resource )
Plane 자동 수행 확인 > 하단 Plan Details 클릭 확인
서버 모니터링
watch -d tree .atlantis
root@Atlantis:~# cat .atlantis/repos/icebreaker70/t101-cicd/1/default/main.tf
resource "null_resource" "example" {}
--atlantis-url="$URL" \
--gh-user="$USERNAME" \
--gh-token="$TOKEN" \
--gh-webhook-secret="$SECRET" \
--repo-allowlist="$REPO_ALLOWLIST"
{"level":"info","ts":"2024-07-13T07:01:11.924Z","caller":"server/server.go:319","msg":"Supported VCS Hosts%!(EXTRA string=hosts, []models.VCSHostType=[Github])","json":{}}
{"level":"info","ts":"2024-07-13T07:01:12.382Z","caller":"server/server.go:467","msg":"Utilizing BoltDB","json":{}}
{"level":"info","ts":"2024-07-13T07:01:12.392Z","caller":"policy/conftest_client.go:153","msg":"failed to get default conftest version. Will attempt request scoped lazy loads DEFAULT_CONFTEST_VERSION not set","json":{}}
{"level":"info","ts":"2024-07-13T07:01:12.393Z","caller":"server/server.go:1017","msg":"Atlantis started - listening on port 4141","json":{}}
{"level":"info","ts":"2024-07-13T07:01:12.393Z","caller":"scheduled/executor_service.go:51","msg":"Scheduled Executor Service started","json":{}}
{"level":"info","ts":"2024-07-13T07:10:02.353Z","caller":"server/server.go:1094","msg":"Apply Lock: {false true 0001-01-01 00:00:00 +0000 UTC }","json":{}}
{"level":"info","ts":"2024-07-13T08:00:44.613Z","caller":"events/working_dir.go:235","msg":"creating dir '/root/.atlantis/repos/icebreaker70/t101-cicd/1/default'","json":{"repo":"icebreaker70/t101-cicd","pull":"1"}}
{"level":"info","ts":"2024-07-13T08:00:45.774Z","caller":"events/project_command_builder.go:495","msg":"found no atlantis.yaml file","json":{"repo":"icebreaker70/t101-cicd","pull":"1"}}
{"level":"info","ts":"2024-07-13T08:00:45.774Z","caller":"events/project_finder.go:147","msg":"filtered modified files to 1 file(s) in the autoplan file list: [main.tf]","json":{"repo":"icebreaker70/t101-cicd","pull":"1"}}
{"level":"info","ts":"2024-07-13T08:00:45.775Z","caller":"events/project_finder.go:176","msg":"there are 1 modified project(s) at path(s): .","json":{"repo":"icebreaker70/t101-cicd","pull":"1"}}
{"level":"info","ts":"2024-07-13T08:00:45.775Z","caller":"events/project_command_builder.go:517","msg":"automatically determined that there were 1 additional projects modified in this pull request: [repofullname=icebreaker70/t101-cicd path=.]","json":{"repo":"icebreaker70/t101-cicd","pull":"1"}}
{"level":"info","ts":"2024-07-13T08:00:45.775Z","caller":"events/project_finder.go:79","msg":"looking for Terraform Cloud workspace from configuration in \"/root/.atlantis/repos/icebreaker70/t101-cicd/1/default\"","json":{"repo":"icebreaker70/t101-cicd","pull":"1"}}
{"level":"info","ts":"2024-07-13T08:00:45.777Z","caller":"terraform/terraform_client.go:305","msg":"cannot determine which version to use from terraform configuration, detected 0 possibilities.","json":{"repo":"icebreaker70/t101-cicd","pull":"1"}}
{"level":"info","ts":"2024-07-13T08:00:45.777Z","caller":"vcs/instrumented_client.go:218","msg":"updating vcs status","json":{"repo":"icebreaker70/t101-cicd","pull":"1"}}
{"level":"info","ts":"2024-07-13T08:00:46.113Z","caller":"vcs/instrumented_client.go:218","msg":"updating vcs status","json":{"repo":"icebreaker70/t101-cicd","pull":"1"}}
{"level":"info","ts":"2024-07-13T08:00:46.415Z","caller":"events/project_locker.go:86","msg":"acquired lock with id \"icebreaker70/t101-cicd/./default\"","json":{"repo":"icebreaker70/t101-cicd","pull":"1"}}
{"level":"info","ts":"2024-07-13T08:00:46.907Z","caller":"models/shell_command_runner.go:161","msg":"successfully ran \"/usr/bin/terraform init -input=false -upgrade\" in \"/root/.atlantis/repos/icebreaker70/t101-cicd/1/default\"","json":{"repo":"icebreaker70/t101-cicd","pull":"1","duration":0.486750957}}
{"level":"info","ts":"2024-07-13T08:00:46.949Z","caller":"terraform/terraform_client.go:412","msg":"successfully ran \"/usr/bin/terraform workspace show\" in \"/root/.atlantis/repos/icebreaker70/t101-cicd/1/default\"","json":{"repo":"icebreaker70/t101-cicd","pull":"1","duration":0.042463857}}
{"level":"info","ts":"2024-07-13T08:00:47.255Z","caller":"models/shell_command_runner.go:161","msg":"successfully ran \"/usr/bin/terraform plan -input=false -refresh -out \\\"/root/.atlantis/repos/icebreaker70/t101-cicd/1/default/default.tfplan\\\"\" in \"/root/.atlantis/repos/icebreaker70/t101-cicd/1/default\"","json":{"repo":"icebreaker70/t101-cicd","pull":"1","duration":0.305521429}}
{"level":"info","ts":"2024-07-13T08:00:47.255Z","caller":"vcs/instrumented_client.go:218","msg":"updating vcs status","json":{"repo":"icebreaker70/t101-cicd","pull":"1"}}
{"level":"info","ts":"2024-07-13T08:00:47.602Z","caller":"events/instrumented_project_command_runner.go:88","msg":"plan success. output available at: https://github.com/icebreaker70/t101-cicd/pull/1","json":{"repo":"icebreaker70/t101-cicd","pull":"1"}}
{"level":"info","ts":"2024-07-13T08:00:48.185Z","caller":"vcs/instrumented_client.go:218","msg":"updating vcs status","json":{"repo":"icebreaker70/t101-cicd","pull":"1"}}
atlantis help
cat /etc/passwd
atlantis apply -d . && cat /etc/passwd
Atlantis Web Console
Add a comment apply > apply 결과 확인
atlantis apply -d .
정상적으로 test 브랜치가 Merge 되었으면 삭제 처리
![](https://velog.velcdn.com/images/sejkim/post/d3afa7df-a536-472e-9389-73de352d3f6f/image.png)
Atlantis repos에 1 디렉토리 삭제 됨
웹 확인
Local Git
$ git checkout main
Switched to branch 'main'
Your branch is up to date with 'origin/main'.
$ ls
README.md
$ git pull
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (1/1), 903 bytes | 451.00 KiB/s, done.
From https://github.com/icebreaker70/t101-cicd
ced2cc1..c0f445e main -> origin/main
Updating ced2cc1..c0f445e
Fast-forward
main.tf | 1 +
1 file changed, 1 insertion(+)
create mode 100644 main.tf
$ ls
README.md main.tf
$ cat main.tf
resource "null_resource" "example" {}
$ git log
commit c0f445ee09513e1be2ffb1a01eb987f00330e7a3 (HEAD -> main, origin/main, origin/HEAD)
Merge: ced2cc1 ff7416c
Author: KimSeongJung <41236393+icebreaker70@users.noreply.github.com>
Date: Sat Jul 13 17:36:16 2024 +0900
Merge pull request #1 from icebreaker70/test
create null resource
commit ff7416c9f5d45b94f462901ce4b735dc67edbb7e (origin/test, test)
Author: Kim Seong Jung <icebreaker70@gmail.com>
Date: Sat Jul 13 16:43:26 2024 +0900
add main.tf
commit ced2cc193adbef3233b93bee198f22c6a60de3c6
Author: KimSeongJung <41236393+icebreaker70@users.noreply.github.com>
Date: Sat Jul 13 11:57:29 2024 +0900
Initial commit
Create a pull request so you can test Atlantis.
# [TIP] You could add a null resource as a test:
resource "null_resource" "example" {}
# Or just modify the whitespace in a file.
Autoplan
You should see Atlantis logging about receiving the webhook and you should see the output of
terraform plan
on your repo.
Atlantis tries to figure out the directory to plan in based on the files modified.
If you need to customize the directories that Atlantis runs in or the commands it runs if you're using workspaces or
.tfvars
files, see atlantis.yaml Reference.
Manual Plan
To manually
plan
in a specific directory or workspace, comment on the pull request using the-d
or-w
flags:
atlantis plan **-d** mydir # 디렉터리
atlantis plan **-w** staging # 워크스페이스
To add additional arguments to the underlying
terraform plan
you can use:
atlantis plan -- -**target**=resource -**var** 'foo=bar'
Apply
If you'd like to
apply
, type a comment:atlantis apply
. You can use the-d
or-w
flags to point Atlantis at a specific plan.
Otherwise it tries to apply the plan for the root directory.
$ git branch vpc && git checkout vpc && git branch
Switched to branch 'vpc'
main
test
* vpc
terraform {
required_version = ">= 1.8"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.34"
}
helm = {
source = "hashicorp/helm"
version = ">= 2.9"
}
}
}
provider "aws" {
region = local.region
}
# Required for public ECR where Karpenter artifacts are hosted
provider "aws" {
region = "us-east-1"
alias = "virginia"
}
provider "helm" {
kubernetes {
host = module.eks.cluster_endpoint
cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "aws"
# This requires the awscli to be installed locally where Terraform is executed
args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
}
}
}
################################################################################
# Common data/locals
################################################################################
data "aws_ecrpublic_authorization_token" "token" {
provider = aws.virginia
}
data "aws_availability_zones" "available" {}
locals {
name = "ex-${basename(path.cwd)}"
region = "ap-northeast-2"
vpc_cidr = "10.0.0.0/16"
azs = slice(data.aws_availability_zones.available.names, 0, 3)
tags = {
Blueprint = local.name
GithubRepo = "github.com/aws-ia/terraform-aws-eks-blueprints"
}
}
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "~> 5.0"
name = local.name
cidr = local.vpc_cidr
azs = local.azs
private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)]
public_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 48)]
enable_nat_gateway = true
single_nat_gateway = true
public_subnet_tags = {
"kubernetes.io/role/elb" = 1
}
private_subnet_tags = {
"kubernetes.io/role/internal-elb" = 1
# Tags subnets for Karpenter auto-discovery
"karpenter.sh/discovery" = local.name
}
tags = local.tags
}
$ git status
On branch vpc
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: main.tf
Untracked files:
(use "git add <file>..." to include in what will be committed)
vpc.tf
no changes added to commit (use "git add" and/or "git commit -a")
$ git add .
$ git commit -m "vpc 코드 작성"
[vpc 1486323] vpc 코드 작성
2 files changed, 88 insertions(+), 1 deletion(-)
create mode 100644 vpc.tf
$ git push origin vpc
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 8 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 1.25 KiB | 1.25 MiB/s, done.
Total 4 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote:
remote: Create a pull request for 'vpc' on GitHub by visiting:
remote: https://github.com/icebreaker70/t101-cicd/pull/new/vpc
remote:
To https://github.com/icebreaker70/t101-cicd
* [new branch] vpc -> vpc
Create a pull request
atlantis plan 결과
Atlantis .atlantis 모니터링
Atlantis web console 화면
atlantis apply 실행화면
vpc 화면
$ git checkout main
$ git branch -D vpc
$ git branch eks && git checkout eks && git branch
Switched to branch 'eks'
* eks
main
Cluster
################################################################################
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "~> 20.11"
cluster_name = local.name
cluster_version = "1.30"
# Give the Terraform identity admin access to the cluster
# which will allow it to deploy resources into the cluster
enable_cluster_creator_admin_permissions = true
cluster_endpoint_public_access = true
cluster_addons = {
coredns = {
configuration_values = jsonencode({
tolerations = [
# Allow CoreDNS to run on the same nodes as the Karpenter controller
# for use during cluster creation when Karpenter nodes do not yet exist
{
key = "karpenter.sh/controller"
value = "true"
effect = "NoSchedule"
}
]
})
}
eks-pod-identity-agent = {}
kube-proxy = {}
vpc-cni = {}
}
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets
eks_managed_node_groups = {
karpenter = {
instance_types = ["t3.medium"]
min_size = 2
max_size = 3
desired_size = 2
labels = {
# Used to ensure Karpenter runs on nodes that it does not manage
"karpenter.sh/controller" = "true"
}
taints = {
# The pods that do not tolerate this taint should run on nodes
# created by Karpenter
karpenter = {
key = "karpenter.sh/controller"
value = "true"
effect = "NO_SCHEDULE"
}
}
}
}
tags = merge(local.tags, {
# NOTE - if creating multiple security groups with this module, only tag the
# security group that Karpenter should utilize with the following tag
# (i.e. - at most, only one security group should have this tag in your account)
"karpenter.sh/discovery" = local.name
})
}
output "configure_kubectl" {
description = "Configure kubectl: make sure you're logged in with the correct AWS profile and run the following command to update your kubeconfig"
value = "aws eks --region ${local.region} update-kubeconfig --name ${module.eks.cluster_name}"
}
################################################################################
# Controller & Node IAM roles, SQS Queue, Eventbridge Rules
################################################################################
module "karpenter" {
source = "terraform-aws-modules/eks/aws//modules/karpenter"
version = "~> 20.11"
cluster_name = module.eks.cluster_name
# Name needs to match role name passed to the EC2NodeClass
node_iam_role_use_name_prefix = false
node_iam_role_name = local.name
create_pod_identity_association = true
tags = local.tags
}
################################################################################
# Helm charts
################################################################################
resource "helm_release" "karpenter" {
namespace = "kube-system"
name = "karpenter"
repository = "oci://public.ecr.aws/karpenter"
repository_username = data.aws_ecrpublic_authorization_token.token.user_name
repository_password = data.aws_ecrpublic_authorization_token.token.password
chart = "karpenter"
version = "0.36.2"
wait = false
values = [
<<-EOT
nodeSelector:
karpenter.sh/controller: 'true'
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- key: karpenter.sh/controller
operator: Exists
effect: NoSchedule
settings:
clusterName: ${module.eks.cluster_name}
clusterEndpoint: ${module.eks.cluster_endpoint}
interruptionQueue: ${module.karpenter.queue_name}
EOT
]
lifecycle {
ignore_changes = [
repository_password
]
}
}
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
name: default
spec:
amiFamily: AL2
role: ex-karpenter-mng
subnetSelectorTerms:
- tags:
karpenter.sh/discovery: ex-karpenter-mng
securityGroupSelectorTerms:
- tags:
karpenter.sh/discovery: ex-karpenter-mng
tags:
karpenter.sh/discovery: ex-karpenter-mng
---
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: default
spec:
template:
spec:
nodeClassRef:
name: default
requirements:
- key: "karpenter.k8s.aws/instance-category"
operator: In
values: ["c", "m", "r"]
- key: "karpenter.k8s.aws/instance-cpu"
operator: In
values: ["4", "8", "16", "32"]
- key: "karpenter.k8s.aws/instance-hypervisor"
operator: In
values: ["nitro"]
- key: "karpenter.k8s.aws/instance-generation"
operator: Gt
values: ["2"]
limits:
cpu: 1000
disruption:
consolidationPolicy: WhenEmpty
consolidateAfter: 30s
❯ cat karpenter.yaml
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
name: default
spec:
amiFamily: AL2
role: ex-karpenter-mng
subnetSelectorTerms:
- tags:
karpenter.sh/discovery: ex-karpenter-mng
securityGroupSelectorTerms:
- tags:
karpenter.sh/discovery: ex-karpenter-mng
tags:
karpenter.sh/discovery: ex-karpenter-mng
---
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: default
spec:
template:
spec:
nodeClassRef:
name: default
requirements:
- key: "karpenter.k8s.aws/instance-category"
operator: In
values: ["c", "m", "r"]
- key: "karpenter.k8s.aws/instance-cpu"
operator: In
values: ["4", "8", "16", "32"]
- key: "karpenter.k8s.aws/instance-hypervisor"
operator: In
values: ["nitro"]
- key: "karpenter.k8s.aws/instance-generation"
operator: Gt
values: ["2"]
limits:
cpu: 1000
disruption:
consolidationPolicy: WhenEmpty
consolidateAfter: 30s
apiVersion: apps/v1
kind: Deployment
metadata:
name: inflate
spec:
replicas: 0
selector:
matchLabels:
app: inflate
template:
metadata:
labels:
app: inflate
spec:
terminationGracePeriodSeconds: 0
containers:
- name: inflate
image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
resources:
requests:
cpu: 1
$ git status
On branch eks
Untracked files:
(use "git add <file>..." to include in what will be committed)
eks.tf
example.yaml
karpenter.yaml
nothing added to commit but untracked files present (use "git add" to track)
$ git add .
$ git status
On branch eks
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
new file: eks.tf
new file: example.yaml
new file: karpenter.yaml
$ git commit -m "eks & karpenter 코드 작성"
[eks 271a35d] eks & karpenter 코드 작성
3 files changed, 196 insertions(+)
create mode 100644 eks.tf
create mode 100644 example.yaml
create mode 100644 karpenter.yaml
$ git status
On branch eks
nothing to commit, working tree clean
$ git push origin eks
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 8 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 2.43 KiB | 2.43 MiB/s, done.
Total 5 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote:
remote: Create a pull request for 'eks' on GitHub by visiting:
remote: https://github.com/icebreaker70/t101-cicd/pull/new/eks
remote:
To https://github.com/icebreaker70/t101-cicd.git
* [new branch] eks -> eks
Create a pull request
atlantis plan 결과
Atlantis .atlantis 모니터링
Atlantis web console 화면
atlantis apply 실행화면
eks 화면
$ git checkout main
Switched to branch 'main'
Your branch is behind 'origin/main' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
$ git pull
Updating aca0787..0ab5645
Fast-forward
eks.tf | 131 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
example.yaml | 21 +++++++++++++++++++++
karpenter.yaml | 44 ++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 196 insertions(+)
create mode 100644 eks.tf
create mode 100644 example.yaml
create mode 100644 karpenter.yaml
$ git branch -D eks
Deleted branch eks (was 271a35d).
$ git log
commit 0ab5645320cfad87a3e88ef65f8c24c4a3ae67ee (HEAD -> main, origin/main, origin/HEAD)
Merge: aca0787 271a35d
Author: KimSeongJung <41236393+icebreaker70@users.noreply.github.com>
Date: Sat Jul 13 21:50:08 2024 +0900
Merge pull request #3 from icebreaker70/eks
eks & karpenter 코드 작성
commit 271a35dbb9708af88c69e9e6d828fb63e9d90a4d (origin/eks)
Author: Kim Seong Jung <icebreaker70@gmail.com>
Date: Sat Jul 13 21:15:57 2024 +0900
eks & karpenter 코드 작성
commit aca07874f1f06a73d7bd34a94ec9b6fc1bffa04e
Merge: c0f445e 1486323
Author: KimSeongJung <41236393+icebreaker70@users.noreply.github.com>
Date: Sat Jul 13 20:36:56 2024 +0900
Merge pull request #2 from icebreaker70/vpc
vpc 코드 작성
commit 1486323991254bbf2ea5ea1e90a079ba43120fca
Author: Kim Seong Jung <icebreaker70@gmail.com>
Date: Sat Jul 13 20:17:02 2024 +0900
vpc 코드 작성
commit c0f445ee09513e1be2ffb1a01eb987f00330e7a3
Merge: ced2cc1 ff7416c
Author: KimSeongJung <41236393+icebreaker70@users.noreply.github.com>
Date: Sat Jul 13 17:36:16 2024 +0900
Merge pull request #1 from icebreaker70/test
create null resource
commit ff7416c9f5d45b94f462901ce4b735dc67edbb7e
Author: Kim Seong Jung <icebreaker70@gmail.com>
Date: Sat Jul 13 16:43:26 2024 +0900
add main.tf
commit ced2cc193adbef3233b93bee198f22c6a60de3c6
Author: KimSeongJung <41236393+icebreaker70@users.noreply.github.com>
Date: Sat Jul 13 11:57:29 2024 +0900
Initial commit
Atlantis 배포된 자원은 PR을 통해 삭제 처리 필요함
$ git status
On branch destoryAtlantis
Untracked files:
(use "git add <file>..." to include in what will be committed)
deleted: README.md
deleted: eks.tf
deleted: example.yaml
deleted: karpenter.yaml
deleted: main.tf
deleted: vpc.tf
nothing added to commit but untracked files present (use "git add" to track)
$ git add .
$ git commit -m "destory eks & vpc"
[destoryAtlantis 942e488] destory eks & vpc
5 files changed, 224 deletions(-)
delete mode 100644 README.md
delete mode 100644 eks.tf
delete mode 100644 example.yaml
delete mode 100644 karpenter.yaml
delete mode 100644 vpc.tf
$ git status
On branch destoryAtlantis
nothing to commit, working tree clean
$ git push origin destoryAtlantis
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 906 bytes | 906.00 KiB/s, done.
Total 3 (delta 0), reused 1 (delta 0), pack-reused 0 (from 0)
To https://github.com/icebreaker70/t101-cicd.git
da8fae8..942e488 destoryAtlantis -> destoryAtlantis
$ git checkout main
Switched to branch 'main'
Your branch is behind 'origin/main' by 3 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
$ git branch -D destoryAtlantis
Deleted branch destoryAtlantis (was 942e488).
# URL 변수 지정
URL="http://$(curl -s ipinfo.io/ip):4141"
echo $URL