MLOps 구축기 - Pipeline

pose estimation을 위한 pipeline입니다.

1. Data Loader .py

import os
import cv2
import numpy as np
import argparse
 
def load_data(data_path):
"""
data_path로 부터 video data를 numpy로 Load 
"""
    cap = cv2.VideoCapture(data_path)
    assert cap.isOpened(), f"Faild to load video file {data_path}"
        
    result = []
    while(cap.isOpened()):
        flag, img = cap.read()
        if not flag:
            break
        result.append(img)
    result = np.array(result)
    print(result.shape)
    np.save('demo_data.npy', result)
    return result

if __name__ == '__main__':
    argument_parser = argparse.ArgumentParser()
    
    argument_parser.add_argument(
        '--data_path', type=str,
        help='Input your data path'
    )
    args = argument_parser.parse_args()
    data_path = args.data_path
    
    # Load Data from data_path
    load_data(data_path)

• Test code

python validate_data.py --minio_path="data" \
--data_path="data" --detection_config "./model_weight/detection/yolov3_d53_320_273e_coco.py" \
--detection_weight "./model_weight/detection/yolo/yolov3_d53_320_273e_coco-421362b6.pth"

python prepare_data.py --minio_img_path="../2_validate_data/raw_data.npz"\
--minio_person_path="../2_validate_data/person_data.parquet" \
--local_save_path="./data" \
--img_path="./image" \
--pose_config="model_weight/mmpose_2d_weight/hrnet_w48_coco_256x192.py" \
--pose_weight="model_weight/mmpose_2d_weight/hrnet_w48_coco_256x192-b9e0b3ab_20200708.pth"

2. Dockerfile

FROM python:3.10
RUN mkdir /code
WORKDIR /code
COPY . /code/
RUN pip install opencv-python
RUN pip install opencv-python-headless
ENTRYPOINT ["python", "data_loader.py"]
CMD ["--data_path", "/code/data/demo_data.mp4"]

3. Deploy

• object detection 배포

  ## data_loader
  $ docker build -t juliy9812/detection_load_data:latest .
  $ docker push juliy9812/detection_load_data:latest
  
  ## object detection
  $ docker build -t juliy9812/detection_inference:latest .
  $ docker push juliy9812/detection_inference:latest

• vision transformer 배포

  $ docker build --no-cache -t juliy9812/transformer_inference:latest .
  $ docker push juliy9812/transformer_inference:latest

4. pipeline 생성

• pipeline.py 실행
• 실행 이후 만들어진 yaml/tar.gz 파일을 kubeflow central dashboard pipeline에서 upload