wsl2로 우분투 깔아서 airflow 설치

MUUU·2022년 3월 3일

프로젝트

목록 보기

4/6

wsl2 설치 : 윈도우에서도 ubutu로 개발가능

https://docs.microsoft.com/ko-kr/windows/wsl/install

*ubuntu 18.04LTS : python 3.6
---- pip install error 발생 pip 3.7이상부터 지원
( 검색해보면 18.04 LTS로 airflow 잘 실행하던데 왜 안되는지 모르겠따.... )
ubuntu 20.04LTS : 제대로 됨 python 3.8

*name/password: admin1234 admin1234

msi 파일 설치하고 컴퓨터 재부팅

microsoft store에서 windows terminal 설치 : 이걸로 작업할 예정

windows terminal 웬만하면 관리자권한으로 실행
wsl -l -v # 실행중인 버전 확인
wsl --set-default-version [linux 배포이름] 2 # wsl version 1 to 2 change (나는 Ubuntu-20.04로 설정)
wsl.exe
wsl pwd

wsl --list --online

Ubuntu          Ubuntu
Debian          Debian GNU/Linux
kali-linux      Kali Linux Rolling
openSUSE-42     openSUSE Leap 42
SLES-12         SUSE Linux Enterprise Server v12
Ubuntu-16.04    Ubuntu 16.04 LTS
Ubuntu-18.04    Ubuntu 18.04 LTS
Ubuntu-20.04    Ubuntu 20.04 LTS
wsl --install -d ubuntu-18.04
( 뭐가 뜬다)```

wsl -l -v #버전확인
시작
wsl.exe -d Ubuntu-20.04

windows terminal 설정에서 기본 프로필 변경

airflow 설치

https://lsjsj92.tistory.com/631

python3
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3 get-pip.py

sudo apt install python-pip
pip install --upgrade pip
sudo python3 -m pip uninstall pip

pip 설치에러
https://musclebear.tistory.com/131
https://jungwoon.github.io/airflow/2019/02/26/Airflow.html

** 나는 pip 때문에 고생했다.. 삽질하다보니까 깔렸따....

pip install 'apache-airflow==2.2.4' \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.4/constraints-3.7.txt"

-warning :

WARNING: The script wheel is installed in '/home/admin1234/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts pip, pip3 and pip3.8 are installed in '/home/admin1234/.local/bin' which is not on PATH.

export PATH=" $HOME/.local/bin:$ PATH"

pip install apache-airflow
airflow db init ##필수다... 안 하면 oops 오류 뜬다

airflow sheduler
airflow webserver -p 8080 #로그인 창 뜨면 성공

airflow users create \
--username admin1234 \
--firstname FIRST_NAME \
--lastname LAST_NAME \
--role Admin \
--password admin1234 \
--email bommme13@gmail.com

-지저분한 ui인데 airflow.cfg 파일의 load_examples=False로 설정하면 없어진다고 함...

history : 삽질흔적,,,

 	1	pwd
    2  ubuntu\
    3  ubuntu --version
    4  --version
    5  cat/etc/issue
    6  lsb_reales-a
    7  uname-m
    8  uname -m
    9  cat /etc/*release
   10  cd c
   11  cd ..
   12  cd jupyter
   13  cd 태블로_파이널PJ/
   14  airflow_test/
   15  sudo apt install python-pip
   16  curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
   17  python2 get-pip.py
   18  python3 get-pip.py
   19  pip apache-airflow
   20  pip
   21  export PATH="$HOME/.local/bin:$PATH"
   22  pip apache-airflow
   23  airflow
   24  pip install apache-airflow
   25  airflow
   26  dir
   27  pip uninstall apache-airflow
   28  cd airflow-test
   29  dir
   30  cd airflow_test
   31  pip install apache-airflow
   32  dir
   33  airflow webserver
   34  python3 --version
   35  sudo apt-get update
   36  sudp apt-get install python-pip -y
   37  sudo apt-get install python-pip -y
   38  python-pip.py
   39  python-pip
   40  pip install
   41  sudo pip install --upgrade pip
   42  sudo apt-get install python-setuptools -y
   43  sudo apt-get install python-dev -y
   44  sudo apt-get install python-mysqldb -y
   45  python-mysql
   46  sudo apt-get pip install --upgrade
   47  sudo pip --upgrade
   48  sudo apt-get install postgresql postgresql-contrib -y
   49  sudo -u postgres psql
   50  sudo -u postgres sql
   51  sudo -u postgresql
   52  sudo -u psql
   53  sudo -u postgresql psql
   54  sudo -u postgres psql
   55  pip install apache-airflow==1.10.12  --constraint " https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt "
   56  pip install apache-airflow==1.10.12  --constraint " https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt "
   57  airflow webserver
   58  pip install 'apache-airflow==2.2.4'  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.4/constraints-3.7.txt"
   59  airflow webserver
   60  airflow initdb
   61  airflow webserver -p 8080
   62  airflow scheduler
   63  airflow webserver
   64  pip --version
   65  python3 -m pip install apache-airflow
   66  airflow users create --username admin --firstname airflow --lastname your_lastname --role Admin --email airflow@sample.com -p your_password
   67  username
   68  screen -dmS airflow-webserver airflow webserver -p 8080
   69  airflow webserver
   70  airflow db init
   71  airflow webserver -p 8080
   72  airflow create_user -r admin -u user_name -e bommme13@gmail.com -f first_name -l last_name -p password
   73  airflow users create  --username admin  --firstname FIRST_NAME  --lastname LAST_NAME  --role Admin  --password admin  --email bommme13@gmail.com
   74  airflow webserver -p 8080
   75  airflow users create  --username admin1234  --firstname FIRST_NAME  --lastname LAST_NAME  --role Admin  --password admin1234  --email bommme13@gmail.com
   76  airflow webserver -p 8080
   77  history

airflow 개념

Webserver: Web UI로 workflow 상태를 표시하고 각종 기능(실행, 재시작, 로그, 권한 관리)을 제공

DAG: workflow를 실행하는 방법을 설명하지만,

Operator: 실제로 수행할 작업을 결정합니다.

Scheduler: 모든 작업과 DAG를 관리
Executor: Task Instance를 실행하는 주체, Worker의 동작방식을 정의
Worker: 실제 작업을 실행하는 주체
Broker: 실행가능 작업들이 들어가는 공간
MetaDB: 파이프라인의 Metadata들(DAG,Task 등)이 저장되어 있는 DB
BashOperator - bash command 실행
PythonOperator - 임의의 파이썬 함수 호출
EmailOperator - email 송신
SimpleHttpOperator - HTTP request 송신
MySqlOperator, SqliteOperator, PostgresOperator, MsSqlOperator, OracleOperator, * JdbcOperator, etc. - SQL command 실행
Sensor - 다음과 같은 것들을 기다리며 확인 (time, file, database row, S3 key, etc…)
Tasks: Operator가 인스턴스화 된 것. 매개 변수화 된 작업은 DAG의 노드가 됩니다.