Airflow 기초 (3) Testing

록스블로그·2023년 12월 7일
0

로컬 환경에서 python airflow-dag.py 로 오류없이 아무 stdout없이 수행이 된다면 정상적으로 문법적인 오류가 없다는 의미이다.
웹서버에서도 확인이 가능하지만
cli 에서는

airflow dags list
dag_id                                   | filepath                                                                                                              | owner   | paused
=========================================+=======================================================================================================================+=========+=======
dataset_consumes_1                       | /home/ubuntu/airflow/env/lib/python3.10/site-packages/airflow/example_dags/example_datasets.py                        | airflow | True
dataset_consumes_1_and_2                 | /home/ubuntu/airflow/env/lib/python3.10/site-packages/airflow/example_dags/example_datasets.py                        | airflow | True
dataset_consumes_1_never_scheduled       | /home/ubuntu/airflow/env/lib/python3.10/site-packages/airflow/example_dags/example_datasets.py                        | airflow | True
dataset_consumes_unknown_never_scheduled | /home/ubuntu/airflow/env/lib/python3.10/site-packages/airflow/example_dags/example_datasets.py                        | airflow | True
dataset_produces_1                       | /home/ubuntu/airflow/env/lib/python3.10/site-packages/airflow/example_dags/example_datasets.py                        | airflow | True
dataset_produces_2                       | /home/ubuntu/airflow/env/lib/python3.10/site-packages/airflow/example_dags/example_datasets.py                        | airflow | True
example_bash_operator                    | /home/ubuntu/airflow/env/lib/python3.10/site-packages/airflow/example_dags/example_bash_operator.py                   | airflow | True
example_branch_datetime_operator         | /home/ubuntu/airflow/env/lib/python3.10/site-packages/airflow/example_dags/example_branch_datetime_operator.py        | airflow | True
example_branch_datetime_operator_2       | /home/ubuntu/airflow/env/lib/python3.10/site-packages/airflow/example_dags/example_branch_datetime_operator.py        | airflow | True
...
airflow tasks list tutorial 
airflow tasks list tutorial --tree
<Task(BashOperator): print_date>
    <Task(BashOperator): sleep>
    <Task(BashOperator): templated>

이런식으로 확인이 충분히 가능하다

airflow tasks test 
airflow dags test

해당 명령은 인스턴스를 로컬로 실행하고 stdout으로 출력하며 mysql이나 성공 실패 등을 디비에 기록하지 않는다.

2023-12-07 12:11:33,250] {taskinstance.py:1087} INFO - Dependencies all met for <TaskInstance: tutorial.print_date __***_temporary_run_2023-12-07T12:11:33.207780+00:00__ [None]>
[2023-12-07 12:11:33,256] {taskinstance.py:1087} INFO - Dependencies all met for <TaskInstance: tutorial.print_date __***_temporary_run_2023-12-07T12:11:33.207780+00:00__ [None]>
[2023-12-07 12:11:33,256] {taskinstance.py:1283} INFO -
--------------------------------------------------------------------------------
[2023-12-07 12:11:33,256] {taskinstance.py:1284} INFO - Starting attempt 1 of 2
[2023-12-07 12:11:33,256] {taskinstance.py:1285} INFO -
--------------------------------------------------------------------------------
[2023-12-07 12:11:33,257] {taskinstance.py:1304} INFO - Executing <Task(BashOperator): print_date> on 2015-06-01T00:00:00+00:00
[2023-12-07 12:11:33,332] {taskinstance.py:1511} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_EMAIL=***@example.com
AIRFLOW_CTX_DAG_OWNER=***
AIRFLOW_CTX_DAG_ID=tutorial
AIRFLOW_CTX_TASK_ID=print_date
AIRFLOW_CTX_EXECUTION_DATE=2015-06-01T00:00:00+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=__***_temporary_run_2023-12-07T12:11:33.207780+00:00__
[2023-12-07 12:11:33,333] {subprocess.py:63} INFO - Tmp dir root location:
 /tmp
[2023-12-07 12:11:33,333] {subprocess.py:75} INFO - Running command: ['/usr/bin/bash', '-c', 'date']
[2023-12-07 12:11:33,340] {subprocess.py:86} INFO - Output:
[2023-12-07 12:11:33,343] {subprocess.py:93} INFO - Thu Dec  7 12:11:33 UTC 2023
[2023-12-07 12:11:33,343] {subprocess.py:97} INFO - Command exited with return code 0
WARNING: MYSQL_OPT_RECONNECT is deprecated and will be removed in a future version.
[2023-12-07 12:11:33,361] {taskinstance.py:1322} INFO - Marking task as SUCCESS. dag_id=tutorial, task_id=print_date, execution_date=20150601T000000, start_date=, end_date=20231207T121133
[2023-12-07 12:14:48,383] {dag.py:3651} INFO - dagrun id: tutorial
/home/ubuntu/airflow/env/lib/python3.10/site-packages/airflow/models/dag.py:3660 RemovedInAirflow3Warning: Calling `DAG.create_dagrun()` without an explicit data interval is deprecated
[2023-12-07 12:14:48,393] {dag.py:3668} INFO - created dagrun <DagRun tutorial @ 2015-06-01T00:00:00+00:00: manual__2015-06-01T00:00:00+00:00, state:running, queued_at: None. externally triggered: False>
[2023-12-07 12:14:48,398] {dagrun.py:606} INFO - Marking run <DagRun tutorial @ 2015-06-01T00:00:00+00:00: manual__2015-06-01T00:00:00+00:00, state:running, queued_at: None. externally triggered: False> successful
[2023-12-07 12:14:48,456] {dagrun.py:657} INFO - DagRun Finished: dag_id=tutorial, execution_date=2015-06-01T00:00:00+00:00, run_id=manual__2015-06-01T00:00:00+00:00, run_start_date=2015-06-01T00:00:00+00:00, run_end_date=2023-12-07 12:14:48.399001+00:00, run_duration=268834488.399001, state=success, external_trigger=False, run_type=manual, data_interval_start=2015-06-01T00:00:00+00:00, data_interval_end=2015-06-02T00:00:00+00:00, dag_hash=None
profile
어려움에 성장하는 데이터 엔지니어

0개의 댓글