Terraform 101 - 4기 (4주차)

가시다(gasida) 님이 진행하는 Terraform T101 4기 실습 스터디 게시글입니다.
책 '테라폼으로 시작하는 IaC' 를 참고하였고, 스터디하면서 도움이 될 내용들을 정리하고자 합니다.

4주차는 3주차때 업무적으로 Amazon MSK 버전업 할 필요성이 있어서 Terraform을 이용하여 VPC와 MSK 생성하는 코드를 작성하여 실행하였는데,

MSK Configuration 삭제 후 생성(Cluster 사용중으로 삭제 실패) 또는 생성 후 삭제(중복된 이름 존재로 생성 실패) 둘다 오류가 발생하였습니다.

임시 방편으로 AWS Console에서 수작업으로 Configuration을 변경 후 실행하는 방식으로 해결하였습니다.
본 글에서는 Configuration 이름을 고정대신 MSK버전 변수와 연결하고 & Life Cycle 옵션을 생성 후 삭제로 변경하여 해결한 사례를 공유하고자 합니다.

Provider 일종인 Packer를 활용하여 Multi-Volumes 지원되는 Custom AMI를 생성하고 생성된 Custom AMI를 이용하여 EC2 생성해 보는 내용을 정리하고자 합니다.

Terraform 코드는 3주차 내용에 수정 및 추가하는 방식으로 진행 하였음을 알려드립니다.

코드 작성시 고려한 사항

1. MSK Cluster와 Configuration 상호 종속성 해소
2. AMI 생성 자동화 Tool인 Packer를 활용하여 Multi-Volume
   $df -h
   Filesystem Size Used Avail Use% Mounted on
   /dev/nvme0n1p1 8.0G 2.3G 5.8G 29% /
   /dev/nvme1n1 5.0G 69M 5.0G 2% /engn
   /dev/nvme2n1 20G 176M 20G 1% /logs
3. Custom-AMI를 이용하여 Terraform으로 EC2 생성

MSK 버전업 시 Cluster와 Configuration간 교착 이슈 해결

코드 수정

module > msk > config.tf

aws_msk_configuration 명을 가변 처리될 수 있도록 수정

- 수정 전

  resource "aws_msk_configuration" "msk_config" {
  for_each       = var.msk_cluster

  name           =  msk-${local.default_tag}-${each.key}-config
  kafka_versions = [each.value.kafka_version]

  lifecycle {
    create_before_destroy = false
  }

  server_properties = <<-PROPERTIES
  auto.create.topics.enable=true
  default.replication.factor=2
  min.insync.replicas=1
  PROPERTIES
  }

- 수정 후

• msk_configuration name을 고정하지 않고 버전 문자열을 붙여 버전업 시 변경될 수 있도록 함
• Lifecycle > create_before_destory = true 명시(기본값은 false) 생성 후 삭제하도록 하여 msk_cluster 의존성을 해소

  resource "aws_msk_configuration" "msk_config" {
  for_each       = var.msk_cluster

  name           = format("msk-${local.default_tag}-${each.key}-config-%s", replace(each.value.kafka_version, ".", "-"))
  kafka_versions = [each.value.kafka_version]

  lifecycle {
    create_before_destroy = true
  }

  server_properties = <<-PROPERTIES
  auto.create.topics.enable=true
  default.replication.factor=2
  min.insync.replicas=1
  PROPERTIES
  }

services > backing > msk.auto.tfvars

• MSK 버전업 위해 kafka_version을 "2.8.1" > "3.5.1" 변경

msk_cluster = {
  "msk" = {
    kafka_version          = "3.5.1"
    number_of_broker_nodes = 2
    instance_type          = "kafka.t3.small"
    ebs_volume_size        = 10
  }
}

코드 실행

• $ terraform plan -out tfplan

data.terraform_remote_state.net: Reading...
module.msk.data.aws_subnets.msk_subnets: Reading...
module.msk.data.aws_vpc.vpc: Reading...
module.msk.aws_cloudwatch_log_group.msk_log_group["msk"]: Refreshing state... [id=/aws/msk/msk-dev-t101-sjkim]
module.msk.aws_msk_configuration.msk_config["msk"]: Refreshing state... [id=arn:aws:kafka:ap-northeast-2:538558617837:configuration/msk-dev-t101-sjkim-msk-config-2-8-1/40d555f5-8cc4-4f37-ba23-9833db7e8198-2]
data.terraform_remote_state.net: Read complete after 0s
module.msk.data.aws_subnets.msk_subnets: Read complete after 0s [id=ap-northeast-2]
module.msk.data.aws_vpc.vpc: Read complete after 0s [id=vpc-0fbfba59d3024823c]
module.msk.aws_security_group.msk_security_group["msk"]: Refreshing state... [id=sg-0bd2b3ff7cb47b2c1]
module.msk.aws_security_group_rule.msk_egress["0.msk.0"]: Refreshing state... [id=sgrule-2375152258]
module.msk.aws_security_group_rule.msk_ingress["4.msk.9098"]: Refreshing state... [id=sgrule-3513788125]
module.msk.aws_security_group_rule.msk_ingress["2.msk.9094"]: Refreshing state... [id=sgrule-2479432527]
module.msk.aws_security_group_rule.msk_ingress["3.msk.9096"]: Refreshing state... [id=sgrule-2365414664]
module.msk.aws_security_group_rule.msk_ingress["0.msk.2182"]: Refreshing state... [id=sgrule-2576964452]
module.msk.aws_security_group_rule.msk_ingress["1.msk.9092"]: Refreshing state... [id=sgrule-2996150662]
module.msk.aws_msk_cluster.msk_cluster["msk"]: Refreshing state... [id=arn:aws:kafka:ap-northeast-2:538558617837:cluster/msk-dev-t101-sjkim-msk-cluster/0e90596e-3783-4795-b74d-0697ef4dbf7d-2]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated
with the following symbols:
  ~ update in-place
+/- create replacement and then destroy

Terraform will perform the following actions:

  # module.msk.aws_msk_cluster.msk_cluster["msk"] will be updated in-place
  ~ resource "aws_msk_cluster" "msk_cluster" {
        id                                            = "arn:aws:kafka:ap-northeast-2:538558617837:cluster/msk-dev-t101-sjkim-msk-cluster/0e90596e-3783-4795-b74d-0697ef4dbf7d-2"
      ~ kafka_version                                 = "2.8.1" -> "3.5.1"
        tags                                          = {
            "Name" = "msk-dev-t101-sjkim-msk-cluster"
        }
        # (20 unchanged attributes hidden)

      ~ configuration_info {
          ~ arn      = "arn:aws:kafka:ap-northeast-2:538558617837:configuration/msk-dev-t101-sjkim-msk-config-2-8-1/40d555f5-8cc4-4f37-ba23-9833db7e8198-2" -> (known after apply)
            # (1 unchanged attribute hidden)
        }

        # (4 unchanged blocks hidden)
    }

  # module.msk.aws_msk_configuration.msk_config["msk"] must be replaced
+/- resource "aws_msk_configuration" "msk_config" {
      ~ arn               = "arn:aws:kafka:ap-northeast-2:538558617837:configuration/msk-dev-t101-sjkim-msk-config-2-8-1/40d555f5-8cc4-4f37-ba23-9833db7e8198-2" -> (known after apply)
      ~ id                = "arn:aws:kafka:ap-northeast-2:538558617837:configuration/msk-dev-t101-sjkim-msk-config-2-8-1/40d555f5-8cc4-4f37-ba23-9833db7e8198-2" -> (known after apply)
      ~ kafka_versions    = [ # forces replacement
          - "2.8.1",
          + "3.5.1",
        ]
      ~ latest_revision   = 3 -> (known after apply)
      ~ name              = "msk-dev-t101-sjkim-msk-config-2-8-1" -> "msk-dev-t101-sjkim-msk-config-3-5-1" # forces replacement
        # (2 unchanged attributes hidden)
    }

Plan: 1 to add, 1 to change, 1 to destroy.

───────────────────────────────────────────────────────
Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

• $ terraform apply tfplan

module.msk.aws_msk_configuration.msk_config["msk"]: Creating...
module.msk.aws_msk_configuration.msk_config["msk"]: Creation complete after 1s [id=arn:aws:kafka:ap-northeast-2:538558617837:configuration/msk-dev-t101-sjkim-msk-config-3-5-1/653d5288-d4f1-42dc-9ac7-20e4858309fa-2]
module.msk.aws_msk_cluster.msk_cluster["msk"]: Modifying... [id=arn:aws:kafka:ap-northeast-2:538558617837:cluster/msk-dev-t101-sjkim-msk-cluster/0e90596e-3783-4795-b74d-0697ef4dbf7d-2]
module.msk.aws_msk_cluster.msk_cluster["msk"]: Still modifying... [id=arn:aws:kafka:ap-northeast-2:5385586178...0e90596e-3783-4795-b74d-0697ef4dbf7d-2, 10s elapsed]

...

module.msk.aws_msk_cluster.msk_cluster["msk"]: Still modifying... [id=arn:aws:kafka:ap-northeast-2:5385586178...0e90596e-3783-4795-b74d-0697ef4dbf7d-2, 1h4m20s elapsed]
module.msk.aws_msk_cluster.msk_cluster["msk"]: Modifications complete after 1h4m26s 

[id=arn:aws:kafka:ap-northeast-2:538558617837:cluster/msk-dev-t101-sjkim-msk-cluster/0e90596e-3783-4795-b74d-0697ef4dbf7d-2]
module.msk.aws_msk_configuration.msk_config["msk"] (deposed object e32c0ee5): Destroying... [id=arn:aws:kafka:ap-northeast-2:538558617837:configuration/msk-dev-t101-sjkim-msk-config-2-8-1/40d555f5-8cc4-4f37-ba23-9833db7e8198-2]
module.msk.aws_msk_configuration.msk_config["msk"]: Destruction complete after 4s

Apply complete! Resources: 1 added, 1 changed, 1 destroyed.

Outputs:

msk = {
  "bootstrap_brokers" = {
    "msk" = "b-1.mskdevt101sjkimms.rdlooj.c2.kafka.ap-northeast-2.amazonaws.com:9092,b-2.mskdevt101sjkimms.rdlooj.c2.kafka.ap-northeast-2.amazonaws.com:9092"
  }
  "bootstrap_brokers_tls" = {
    "msk" = "b-1.mskdevt101sjkimms.rdlooj.c2.kafka.ap-northeast-2.amazonaws.com:9094,b-2.mskdevt101sjkimms.rdlooj.c2.kafka.ap-northeast-2.amazonaws.com:9094"
  }
  "msk_cluster_name" = {
    "msk" = "msk-dev-t101-sjkim-msk-cluster"
  }
  "zookeeper_connect_string" = {
    "msk" = "z-1.mskdevt101sjkimms.rdlooj.c2.kafka.ap-northeast-2.amazonaws.com:2181,z-2.mskdevt101sjkimms.rdlooj.c2.kafka.ap-northeast-2.amazonaws.com:2181,z-3.mskdevt101sjkimms.rdlooj.c2.kafka.ap-northeast-2.amazonaws.com:2181"
  }
}

실행 결과

• $ terraform graph > graph.dot

  msk_cluster는 msk_config를 참조

  

• Amazon MSK > Cluster configurations

  버전업 시작 후 msk-dev-t101-sjkim-msk-config-3-5-1 생성됨
  버전업 완료 후 msk-dev-t101-sjkim-msk-config-2-8-1 삭제됨

  

• Cluster Operations 화면

  1시간 5분 소요

  

• MSK 버전업 된 화면
  

Packer 개요

Packer는 ?

• Packer는 단일 소스 구성에서 여러 플랫폼에 대해 동일한 머신 이미지를 만드는 커뮤니티 도구입니다.
• Packer는 가볍고 모든 주요 운영 체제에서 실행되며 성능이 뛰어나 여러 플랫폼에 대한 머신 이미지를 병렬로 만듭니다.
• Packer는 Chef나 Puppet과 같은 구성 관리를 대체하지 않습니다. 사실, 이미지를 빌드할 때 Packer는 Chef나 Puppet과 같은 도구를 사용하여 이미지에 소프트웨어를 설치할 수 있습니다.
• 머신 이미지는 미리 구성된 운영 체제와 설치된 소프트웨어를 포함하는 단일 정적 단위로, 새로운 실행 머신을 빠르게 만드는 데 사용됩니다.
• 머신 이미지 형식은 플랫폼마다 다릅니다. 몇 가지 예로는 EC2용 AMI , VMware용 VMDK/VMX 파일, VirtualBox용 OVF 내보내기 등이 있습니다.

Packer 설치

macOS

$ brew tap hashicorp/tap
$ brew install hashicorp/tap/packer
$ brew upgrade hashicorp/tap/packer

Redhat / CentOS

$ sudo yum install -y yum-utils
$ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
$ sudo yum -y install packer

Ubuntu

$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
$ sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
$ sudo apt-get update && sudo apt-get install packer

Amazon Linux 2

$ sudo yum install -y yum-utils
$ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo
$ sudo yum -y install packer

Packer 설치결과 확인

$ packer
Usage: packer [--version] [--help] <command> [<args>]

Available commands are:
    build           build image(s) from template
    console         creates a console for testing variable interpolation
    fix             fixes templates from old versions of packer
    fmt             Rewrites HCL2 config files to canonical format
    hcl2_upgrade    transform a JSON template into an HCL2 configuration
    init            Install missing plugins or upgrade plugins
    inspect         see components of a template
    validate        check that a template is valid
    version         Prints the Packer version

Packer 명령어

✔️ Packer init .
테라폼처럼 하단 내용의 플러그인을 다운받게 된다.

required_plugins {
  amazon = { # 아마존 리눅스 이미지를 빌드하기 위해 
    version = ">= 1.3.2" # 현재의 버전 
    source  = "github.com/hashicorp/amazon"
}

✔️ Packer fmt .
Packer 코드의 형식을 맞춰준다. Terraform fmt과 동일하다.

✔️ Packer validate .
Packer 코드의 형식이 유효한지 확인한다. Terraform validate와 동일하다.

✔️ Packer build .
Packer 소스를 이용해 이미지를 빌드한다.

인벤토리의 host를 디폴트로 잡아놓으면 현재 패커에 의해 생성된 인스턴스를 지칭한다.인스턴스를 실행한뒤 인스턴스를 알아서 중단하고 (이미지를 만들기 위해) ssh key, 보안 그룹 등을 만들고 AMI 이미지를 만든다.모든 작업을 실행하고 실행 과정에서 만든 리소스(인스턴스)를 알아서 정리한다.
어떤 vpc에 올려서 작업 할 것인지 지정할 수도 있다.

Terraform과 Packer 이용하여 Custom Ami 생성(Multi-volumes)

Terraform & Packer 코드

• Directory 구조

  services 하위 packer-ami 추가 구성

• backend-packer-ami.tf

terraform {
  required_version = "~> 1.8.5"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.56"
    }

    null = {
      source  = "hashicorp/null"
      version = "~> 3.2.2"
    }

  }

  backend "s3" {
    bucket         = "s3-dev-t101-sjkim-tfstate"
    key            = "terraform/ddb-dev-t101-sjkim-terraform_locks_packer-ami.tfstate"
    region         = "ap-northeast-2"
    dynamodb_table = "ddb-dev-t101-sjkim-terraform_locks_packer-ami"
    encrypt        = "true"
  }
}

provider "aws" {
  region                   = var.region
  shared_credentials_files = ["~/.aws/credentials"]
  profile                  = var.profile

  default_tags {
    tags = {
      Environment      = var.env
      Project          = var.pjt
      Service          = var.svc
      TerraformManaged = true
    }
  }
}

• remote-net.tf

  원격 s3에 저장된 networking tfstate 참조 위해 사용

data "terraform_remote_state" "net" {
  backend = "s3"
  config = {
    bucket = "s3-dev-t101-sjkim-tfstate"
    region = var.region
    key    = "terraform/ddb-dev-t101-sjkim-terraform_locks_net.tfstate"
  }
}

• main.tf

  null_resource 이용하여 packer 설치(macos기준), packer를 호출 함
  packer 호출 시 명령행 옵션으로 변수값을 전달 함

locals {
  timestamp = formatdate("YYYYMMDD-hhmmss", timeadd(timestamp(), "9h"))
  vpc_id    = data.terraform_remote_state.net.outputs.vpc_id
  subnet_id = data.terraform_remote_state.net.outputs.public_subnet_id[0]
  ami_name  = "ami-${var.env}-${var.pjt}-${var.svc}-golden-image-${local.timestamp}"
}

resource "null_resource" "packer_install" {
  triggers = {
    always_run = local.timestamp
  }

  provisioner "local-exec" {
    command = <<-EOT

which packer
if [ $? -ne 0 ]; then
  echo "Packer Install"
  brew install packer
fi

EOT  
  }

}

resource "null_resource" "run_ec2_packer" {
  depends_on = [null_resource.packer_install]
  triggers = {
    always_run = local.timestamp
  }

  provisioner "local-exec" {
    command = <<-EOT
packer version
packer init .
packer fmt .
packer validate .
packer build \
-var 'ami_name=${local.ami_name}' \
-var 'vpc_id=${local.vpc_id}' \
-var 'subnet_id=${local.subnet_id}' \
-var 'env=${var.env}' \
-var 'pjt=${var.pjt}' \
-var 'svc=${var.svc}' \
.

EOT
  }
}

• outputs.tf

output "ami_name" {
  value = local.ami_name
}

• packer_plugin.pkr.hcl

  amazon_ebs Plugin을 설치하도록 함

packer {
  required_plugins {
    amazon = {
      version = ">= 1.3.0"
      source  = "github.com/hashicorp/amazon"
    }
  }
}

• packer_data.pkr.hcl

  Amazon Linux 2의 최신 AMI ID를 가져움

data "amazon-ami" "base-ami" {
  filters = {
    virtualization-type = "hvm"
    name                = "amzn2-ami-kernel-5.10-hvm*"
    root-device-type    = "ebs"
  }

• packer_variable.pkr.hcl

  Packer에서 사용할 변수 값 들을 지정, Packer 명령어 실행 시 변수명과 일치하게 선언 필요 함

variable "product_type" {
  default = {
    ec2               = "ec2"
    ec2_instance_type = "t3.nano"
  }
}

variable "ami_name" {
  type    = string
  default = "ami-env-pjt-svc-golden-iage-yyyymmdd-hhmmss"
}

variable "aws_region" {
  type    = string
  default = "ap-northeast-2"
}

variable "env" {
  type    = string
  default = ""
}

variable "pjt" {
  type    = string
  default = ""
}

variable "svc" {
  type    = string
  default = ""
}

variable "vpc_id" {
  type    = string
  default = ""
}

variable "subnet_id" {
  type    = string
  default = ""
}

• packer_main.pkr.hcl

  AWS EC2 Custom AMI 생성하기 위한 Packer Manifest
  Terraform null_resource에서 packer 실행시 전달 된 인수들을 이용하여 Custom AMI 생성
  provisioner "shell" 로 shell Script 주입 및 실행 가능 함

source "amazon-ebs" "ec2-golden-ami" {
  source_ami        = data.amazon-ami.base-ami.id
  ami_name          = var.ami_name
  region            = var.aws_region
  availability_zone = "${var.aws_region}a"
  instance_type     = var.product_type.ec2_instance_type
  vpc_id            = var.vpc_id
  subnet_id         = var.subnet_id
  ssh_username      = "ec2-user"

  communicator                 = "ssh"
  ssh_clear_authorized_keys    = true
  ssh_pty                      = true
  ssh_disable_agent_forwarding = true

  user_data = <<-EOT
  #!/bin/bash
  sudo ln -sf /usr/share/zoneinfo/Asia/Seoul /etc/localtime
  sudo yum update -y && sudo yum upgrade

  sudo yum remove awscli -y
  curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
  unzip awscliv2.zip
  sudo ./aws/install
  EOT

  temporary_iam_instance_profile_policy_document {
    Version = "2012-10-17"
    Statement {
      Action   = ["ec2:*"]
      Effect   = "Allow"
      Resource = ["*"]
    }
  }

  tags = {
    Name    = var.ami_name
    Env     = var.env
    Project = var.pjt
    Service = var.svc
  }

  launch_block_device_mappings {
    device_name           = "/dev/xvda"
    volume_size           = 8
    volume_type           = "gp3"
    iops                  = 3000
    throughput            = 125
    delete_on_termination = true
    encrypted             = true
  }
  launch_block_device_mappings {
    device_name           = "/dev/xvdh"
    volume_size           = 5
    volume_type           = "gp3"
    iops                  = 3000
    throughput            = 125
    delete_on_termination = true
    encrypted             = true
  }
  launch_block_device_mappings {
    device_name           = "/dev/xvdi"
    volume_size           = 20
    volume_type           = "gp3"
    iops                  = 3000
    throughput            = 125
    delete_on_termination = true
    encrypted             = true
  }
}

build {
  name = "ec2_packer"
  sources = [
    "source.amazon-ebs.ec2-golden-ami"
  ]

  provisioner "shell" {
    inline = ["mkdir ~/tmp"]
  }

  provisioner "shell" {
    script = "./Scripts/default.sh"
  }

  provisioner "shell" {
    script = "./Scripts/user-data-ebs-volume-attach.sh"
  }

  post-processor "manifest" {
    output     = "manifest.json"
    strip_path = true
    custom_data = {
      source_ami_name = "${build.SourceAMIName}"
    }
  }
}

• Scripts > default.sh

  amazon-ssm-agent 설치
  Timezone을 UTC에서 KST로 변경
  History Timestamp 추가

#!/bin/bash
sudo yum install amazon-ssm-agent -y
sudo systemctl enable amazon-ssm-agent
sudo systemctl start amazon-ssm-agent

sudo timedatectl set-timezone Asia/Seoul

PFPATH=/etc/profile
if [ `cat ${PFPATH} | grep "HISTTIMEFORMAT" | wc -l ` -eq "0" ]; then
  {
    echo "###################"
    echo "#history timestamp#"
    echo "HISTTIMEFORMAT='%F %T '"
    echo "export HISTTIMEFORMAT"
  } | sudo tee -a ${PFPATH}
fi
source ${PFPATH}

• Scripts > user-data-ebs-volume-attach.sh

  추가된 소스, 로그 EBS 볼륨을 xfs 파일시스템 생성 후 자동마운되도록 함

#!/bin/bash
sudo yum install xfsprogs -y

sudo lsblk -f
sudo blkid

sudo mkfs -t xfs /dev/xvdh
sudo mkfs -t xfs /dev/xvdi

sudo mkdir /engn /logs
sudo mount /dev/xvdh /engn
sudo mount /dev/xvdi /logs

BLK_ID1=$(sudo blkid /dev/xvdh | cut -f2 -d" ")
BLK_ID2=$(sudo blkid /dev/xvdi | cut -f2 -d" ")

if [[ -z $BLK_ID1 && -z $BLK_ID2 ]]; then
  echo "No blockid  found..."
  exit 1
fi

echo "$BLK_ID1   /engn xfs    defaults   0   2" | sudo tee --append /etc/fstab
echo "$BLK_ID2   /logs xfs    defaults   0   2" | sudo tee --append /etc/fstab

sudo mount -a
echo "Bootstrapping Complete!"%

코드 실행

• $ terraform init && terraform validate && terraform fmt

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- terraform.io/builtin/terraform is built in to Terraform
- Finding hashicorp/aws versions matching "~> 5.56"...
- Finding hashicorp/null versions matching "~> 3.2.2"...
- Installing hashicorp/aws v5.57.0...
- Installed hashicorp/aws v5.57.0 (signed by HashiCorp)
- Installing hashicorp/null v3.2.2...
- Installed hashicorp/null v3.2.2 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Success! The configuration is valid.

• $ terraform plan -out tfplan

null_resource.packer_install: Refreshing state... [id=7883159530147944687]
data.terraform_remote_state.net: Reading...
null_resource.run_ec2_packer: Refreshing state... [id=4648501322306774206]
data.terraform_remote_state.net: Read complete after 0s

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # null_resource.packer_install must be replaced
-/+ resource "null_resource" "packer_install" {
      ~ id       = "7883159530147944687" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "always_run" = "20240706-220023" -> (known after apply)
        }
    }

  # null_resource.run_ec2_packer must be replaced
-/+ resource "null_resource" "run_ec2_packer" {
      ~ id       = "4648501322306774206" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "always_run" = "20240706-220023" -> (known after apply)
        }
    }

Plan: 2 to add, 0 to change, 2 to destroy.

───────────────────────────────────────────────────────
Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

• terraform apply tfpaln

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"
null_resource.run_ec2_packer: Destroying... [id=1210016252837894042]
null_resource.run_ec2_packer: Destruction complete after 0s
null_resource.packer_install: Destroying... [id=6275703667052422767]
null_resource.packer_install: Destruction complete after 0s
null_resource.packer_install: Creating...
null_resource.packer_install: Provisioning with 'local-exec'...
null_resource.packer_install (local-exec): Executing: ["/bin/sh" "-c" "\nwhich packer\nif [ $? -ne 0 ]; then\n  echo \"Packer Install\"\n  brew install packer\nfi\n\n"]
null_resource.packer_install (local-exec): /opt/homebrew/bin/packer
null_resource.packer_install: Creation complete after 0s [id=4211274905702956513]
null_resource.run_ec2_packer: Creating...
null_resource.run_ec2_packer: Provisioning with 'local-exec'...
null_resource.run_ec2_packer (local-exec): Executing: ["/bin/sh" "-c" "packer version\npacker init .\npacker fmt .\npacker validate .\npacker build \\\n-var 'image_name=ami-dev-t101-sjkim-golden-image-20240706-235115' \\\n-var 'vpc_id=vpc-0fbfba59d3024823c' \\\n-var 'subnet_id=subnet-005ab9a3c170e5a8f' \\\n-var 'env=dev' \\\n-var 'pjt=t101' \\\n-var 'svc=sjkim' \\\n.\n\n"]
null_resource.run_ec2_packer (local-exec): Packer v1.11.1
null_resource.run_ec2_packer (local-exec): Error: 2 error(s) occurred:

null_resource.run_ec2_packer (local-exec): * ami_name must be specified
null_resource.run_ec2_packer (local-exec): * ami_name must be between 3 and 128 characters long

null_resource.run_ec2_packer (local-exec):   on packer_main.pkr.hcl line 1:
null_resource.run_ec2_packer (local-exec):   (source code not available)


null_resource.run_ec2_packer (local-exec): ec2_packer.amazon-ebs.ec2-golden-ami: output will be in this color.

null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Prevalidating any provided VPC information
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Prevalidating AMI Name: ami-dev-t101-sjkim-golden-image-20240706-235115
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: Found Image ID: ami-0fd54cba47d6e98dc
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Creating temporary keypair: packer_668959e6-cedf-dcd7-51ce-0617fcd3a965
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Creating temporary security group for this instance: packer_668959e7-fe3e-cc4c-30f2-eaf1c61314ce
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Creating temporary instance profile for this instance: packer-668959e8-4c13-cf3b-c99d-c11c0010ed1c
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Creating temporary role for this instance: packer-668959e8-4c13-cf3b-c99d-c11c0010ed1c
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Attaching policy to the temporary role: packer-668959e8-4c13-cf3b-c99d-c11c0010ed1c
null_resource.run_ec2_packer: Still creating... [10s elapsed]
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Launching a source AWS instance...
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: Instance ID: i-06a59c7325082fb1d
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Waiting for instance (i-06a59c7325082fb1d) to become ready...
null_resource.run_ec2_packer: Still creating... [20s elapsed]
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Using SSH communicator to connect: 52.78.154.183
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Waiting for SSH to become available...
null_resource.run_ec2_packer: Still creating... [30s elapsed]
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Connected to SSH!
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Provisioning with shell script: /var/folders/7r/k37w336504d01lg8qbmg2kxw0000gn/T/packer-shell738009515
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Provisioning with shell script: ./Scripts/default.sh
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: Existing lock /var/run/yum.pid: another copy is running as pid 2457.
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: Another app is currently holding the yum lock; waiting for it to exit...
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami:   The other application is: yum

...

###################
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: Package xfsprogs-5.0.0-10.amzn2.0.1.x86_64 already installed and latest version
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: Nothing to do
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: NAME          FSTYPE LABEL UUID                                 MOUNTPOINT
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: nvme0n1
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: ├─nvme0n1p1   xfs    /     ea0ed322-63d9-4a25-a663-d07aebdedd69 /
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: └─nvme0n1p128
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: nvme2n1
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: nvme1n1
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: /dev/nvme0n1: PTUUID="0fd14126-7130-4d4c-923e-0ac35f04d680" PTTYPE="gpt"
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: /dev/nvme0n1p1: LABEL="/" UUID="ea0ed322-63d9-4a25-a663-d07aebdedd69" TYPE="xfs" PARTLABEL="Linux" PARTUUID="728b980f-595b-451e-b23a-31d7c083e933"
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: /dev/nvme0n1p128: PARTLABEL="BIOS Boot Partition" PARTUUID="7df837a1-36c4-4fca-98ab-ebf657af3ac7"
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: meta-data=/dev/xvdh              isize=512    agcount=8, agsize=163840 blks
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami:          =                       sectsz=512   attr=2, projid32bit=1
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami:          =                       crc=1        finobt=1, sparse=1, rmapbt=0
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami:          =                       reflink=1    bigtime=0 inobtcount=0
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: data     =                       bsize=4096   blocks=1310720, imaxpct=25
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami:          =                       sunit=1      swidth=1 blks
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: log      =internal log           bsize=4096   blocks=2560, version=2
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami:          =                       sectsz=512   sunit=1 blks, lazy-count=1
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: realtime =none                   extsz=4096   blocks=0, rtextents=0
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: meta-data=/dev/xvdi              isize=512    agcount=16, agsize=327680 blks
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami:          =                       sectsz=512   attr=2, projid32bit=1
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami:          =                       crc=1        finobt=1, sparse=1, rmapbt=0
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami:          =                       reflink=1    bigtime=0 inobtcount=0
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: data     =                       bsize=4096   blocks=5242880, imaxpct=25
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami:          =                       sunit=1      swidth=1 blks
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: log      =internal log           bsize=4096   blocks=2560, version=2
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami:          =                       sectsz=512   sunit=1 blks, lazy-count=1
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: realtime =none                   extsz=4096   blocks=0, rtextents=0
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: UUID="bacb028c-3e79-433d-91ec-ca592d5bb509"   /engn xfs    defaults   0   2
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: UUID="26d09664-2b7b-4d95-a222-75f6cb2b202b"   /logs xfs    defaults   0   2
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: Bootstrapping Complete!
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Trying to remove ephemeral keys from authorized_keys files
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Stopping the source instance...
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: Stopping instance
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Waiting for the instance to stop...
null_resource.run_ec2_packer: Still creating... [1m0s elapsed]

...


null_resource.run_ec2_packer: Still creating... [2m0s elapsed]
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Creating AMI ami-dev-t101-sjkim-golden-image-20240706-235115 from instance i-06a59c7325082fb1d
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: AMI: ami-01b17abcd528c6914

...

null_resource.run_ec2_packer: Still creating... [5m10s elapsed]
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Skipping Enable AMI deprecation...
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Adding tags to AMI (ami-01b17abcd528c6914)...
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Tagging snapshot: snap-03d19ad9d21851bc3
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Tagging snapshot: snap-0c91af6d708562966
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Tagging snapshot: snap-099830e0370535945
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Creating AMI tags
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: Adding tag: "Env": "dev"
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: Adding tag: "Name": "ami-dev-t101-sjkim-golden-image-20240706-235115"
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: Adding tag: "Project": "t101"
null_resource.run_ec2_packer (local-exec):     ec2_packer.amazon-ebs.ec2-golden-ami: Adding tag: "Service": "sjkim"
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Creating snapshot tags
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Terminating the source AWS instance...
null_resource.run_ec2_packer: Still creating... [5m20s elapsed]
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Cleaning up any extra volumes...
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: No volumes to clean up, skipping
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Detaching temporary role from instance profile...
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Removing policy from temporary role...
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Deleting temporary role...
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Deleting temporary instance profile...
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Deleting temporary security group...
null_resource.run_ec2_packer: Still creating... [5m30s elapsed]
null_resource.run_ec2_packer (local-exec): ==> ec2_packer.amazon-ebs.ec2-golden-ami: Deleting temporary keypair...
null_resource.run_ec2_packer (local-exec): Build 'ec2_packer.amazon-ebs.ec2-golden-ami' finished after 5 minutes 26 seconds.

null_resource.run_ec2_packer (local-exec): ==> Wait completed after 5 minutes 26 seconds

null_resource.run_ec2_packer (local-exec): ==> Builds finished. The artifacts of successful builds are:
null_resource.run_ec2_packer (local-exec): --> ec2_packer.amazon-ebs.ec2-golden-ami: AMIs were created:
null_resource.run_ec2_packer (local-exec): ap-northeast-2: ami-01b17abcd528c6914

null_resource.run_ec2_packer: Creation complete after 5m32s [id=9194094413005241430]

Apply complete! Resources: 2 added, 0 changed, 2 destroyed.

Outputs:

ami_name = "ami-dev-t101-sjkim-golden-image-20240706-235115"

실행 결과

• Custom AMI 생성위해 EC2 기동 & 스크립트 실행 후 AMI 생성위해 EC2는 중지됨
  

• AMI가 자동으로 생성됨
  

• ec2.tf

resource "aws_instance" "bastion" {
  ami                    = "ami-01b17abcd528c6914"
  instance_type          = "t3.micro"
  vpc_security_group_ids = [aws_security_group.instance.id]
  subnet_id              = data.terraform_remote_state.net.outputs.public_subnet_id[0]
  key_name               = "mykey"

  tags = {
    Name = "ec2-dev-t101-sjkim-bastion"
  }
}


resource "aws_security_group" "instance" {
  name          = var.security_group_name
  vpc_id        = data.terraform_remote_state.net.outputs.vpc_id

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["123.186.158.81/32"]
  }
  egress {
    protocol    = "-1"
    from_port   = 0
    to_port     = 0
    cidr_blocks = ["0.0.0.0/0"]
  }
}

variable "security_group_name" {
  description = "The name of the security group"
  type        = string
  default     = "SG-dev-t101-sjkim-bastion"
}

output "public_ip" {
  value       = aws_instance.bastion.public_ip
  description = "The public IP of the Instance"
}     

• Custom AMI 이용하여 EC2 생성한 결과

  /, /svc, /logs 3개 볼륨 확인되고 자동 마운트 됨
  Timezone이 UTC 대신 Asia/Seoul로 변경됨
  History 명령 실행 시 Timestamp 정보도 같이 보여짐

$ ssh -i ~/keypair/mykey.pem ec2-user@43.203.245.66

The authenticity of host '43.203.245.66 (43.203.245.66)' can't be established.
ED25519 key fingerprint is SHA256:m7TWmfo5sH/zF5RQqucb6Lq1/mWb6HiQy7fbvFXSMRA.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '43.203.245.66' (ED25519) to the list of known hosts.
Last login: Sat Jul  6 23:52:07 2024 from 125.187.158.81
   ,     #_
   ~\_  ####_        Amazon Linux 2
  ~~  \_#####\
  ~~     \###|       AL2 End of Life is 2025-06-30.
  ~~       \#/ ___
   ~~       V~' '->
    ~~~         /    A newer version of Amazon Linux is available!
      ~~._.   _/
         _/ _/       Amazon Linux 2023, GA and supported until 2028-03-15.
       _/m/'           https://aws.amazon.com/linux/amazon-linux-2023/

[ec2-user@ip-172-16-0-17 ~]$ lsblk
NAME          MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1       259:1    0   8G  0 disk 
├─nvme0n1p1   259:3    0   8G  0 part /
└─nvme0n1p128 259:4    0   1M  0 part 
nvme1n1       259:2    0   5G  0 disk /engn
nvme2n1       259:0    0  20G  0 disk /logs

[ec2-user@ip-172-16-0-17 ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        456M     0  456M   0% /dev
tmpfs           464M     0  464M   0% /dev/shm
tmpfs           464M  420K  464M   1% /run
tmpfs           464M     0  464M   0% /sys/fs/cgroup
/dev/nvme0n1p1  8.0G  2.3G  5.8G  28% /
/dev/nvme1n1    5.0G   69M  5.0G   2% /engn
/dev/nvme2n1     20G  176M   20G   1% /logs
tmpfs            93M     0   93M   0% /run/user/1000

[ec2-user@ip-172-16-0-17 ~]$ date
Sun Jul  7 01:07:43 KST 2024

[ec2-user@ip-172-16-0-17 ~]$ history
    1  2024-07-07 01:02:07 blkid
    2  2024-07-07 01:02:23 df -h
    3  2024-07-07 01:02:26 lsblk
    4  2024-07-07 01:05:43 exit
    5  2024-07-07 01:07:43 date
    6  2024-07-07 01:07:46 history