data_lab

Airflow 설치 후 실행

nohup airflow webserver--port 8080 > webserver.log 2>&1 &

Airflow Scheduler 실행

nohup airflow scheduler > scheduler.log 2>&1 &

Aiflow DAG 리스트 확인

airflow dags list

실행 후 결과

dag_id                        | filepath                      | owner   | paused
==============================+===============================+=========+=======
example_bash_operator         | /usr/local/lib/python3.7/site | airflow | False
                              | -packages/airflow/example_dag |         |
                              | s/example_bash_operator.py    |         |
example_branch_datetime_opera | /usr/local/lib/python3.7/site | airflow | True
tor_2                         | -packages/airflow/example_dag |         |
                              | s/example_branch_datetime_ope |         |
                              | rator.py                      |         |
example_branch_dop_operator_v | /usr/local/lib/python3.7/site | airflow | True
3                             | -packages/airflow/example_dag |         |
                              | s/example_branch_python_dop_o |         |
                              | perator_3.py                  |         |
example_branch_labels         | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/example_branch_labels.py    |         |
example_branch_operator       | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/example_branch_operator.py  |         |
example_complex               | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/example_complex.py          |         |
example_dag_decorator         | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/example_dag_decorator.py    |         |
example_external_task_marker_ | /usr/local/lib/python3.7/site | airflow | True
child                         | -packages/airflow/example_dag |         |
                              | s/example_external_task_marke |         |
                              | r_dag.py                      |         |
example_external_task_marker_ | /usr/local/lib/python3.7/site | airflow | True
parent                        | -packages/airflow/example_dag |         |
                              | s/example_external_task_marke |         |
                              | r_dag.py                      |         |
example_nested_branch_dag     | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/example_nested_branch_dag.p |         |
                              | y                             |         |
example_passing_params_via_te | /usr/local/lib/python3.7/site | airflow | True
st_command                    | -packages/airflow/example_dag |         |
                              | s/example_passing_params_via_ |         |
                              | test_command.py               |         |
example_python_operator       | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/example_python_operator.py  |         |
example_short_circuit_operato | /usr/local/lib/python3.7/site | airflow | True
r                             | -packages/airflow/example_dag |         |
                              | s/example_short_circuit_opera |         |
                              | tor.py                        |         |
example_skip_dag              | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/example_skip_dag.py         |         |
example_sla_dag               | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/example_sla_dag.py          |         |
example_subdag_operator       | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/example_subdag_operator.py  |         |
example_subdag_operator.secti | /usr/local/lib/python3.7/site | airflow | True
on-1                          | -packages/airflow/example_dag |         |
                              | s/example_subdag_operator.py  |         |
example_subdag_operator.secti | /usr/local/lib/python3.7/site | airflow | True
on-2                          | -packages/airflow/example_dag |         |
                              | s/example_subdag_operator.py  |         |
example_task_group            | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/example_task_group.py       |         |
example_task_group_decorator  | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/example_task_group_decorato |         |
                              | r.py                          |         |
example_time_delta_sensor_asy | /usr/local/lib/python3.7/site | airflow | True
nc                            | -packages/airflow/example_dag |         |
                              | s/example_time_delta_sensor_a |         |
                              | sync.py                       |         |
example_trigger_controller_da | /usr/local/lib/python3.7/site | airflow | True
g                             | -packages/airflow/example_dag |         |
                              | s/example_trigger_controller_ |         |
                              | dag.py                        |         |
example_trigger_target_dag    | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/example_trigger_target_dag. |         |
                              | py                            |         |
example_weekday_branch_operat | /usr/local/lib/python3.7/site | airflow | True
or                            | -packages/airflow/example_dag |         |
                              | s/example_branch_day_of_week_ |         |
                              | operator.py                   |         |
example_xcom                  | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/example_xcom.py             |         |
example_xcom_args             | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/example_xcomargs.py         |         |
example_xcom_args_with_operat | /usr/local/lib/python3.7/site | airflow | True
ors                           | -packages/airflow/example_dag |         |
                              | s/example_xcomargs.py         |         |
latest_only                   | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/example_latest_only.py      |         |
latest_only_with_trigger      | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/example_latest_only_with_tr |         |
                              | igger.py                      |         |
tutorial                      | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/tutorial.py                 |         |
tutorial_etl_dag              | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/tutorial_etl_dag.py         |         |
tutorial_taskflow_api_etl     | /usr/local/lib/python3.7/site | airflow | True
                              | -packages/airflow/example_dag |         |
                              | s/tutorial_taskflow_api_etl.p |         |
                              | y                             |         |
tutorial_taskflow_api_etl_vir | /usr/local/lib/python3.7/site | airflow | True
tualenv                       | -packages/airflow/example_dag |         |
                              | s/tutorial_taskflow_api_etl_v |         |
                              | irtualenv.py

예시로 들어 있는 airflow dags 위치 알수 있다.

방향성 비순환 그래프 DAG(Directed Acyclic Graph)

apche spark를 접해봤다면, DAG라는 용어가 익숙할 수 있다.
방향성 비순환 그래프라는 말자체 이해하기 좀 어렵지만 그림으로 보면 이해가 쉽다.

Airflow 🪁

최근 데이터 엔지니어 직무 채용공고를 보면 Airflow 사용경험을 우대해 주는 것을 볼 수 있다.

모든 오픈소스를 접해서 공부할 수 없다고 생각하기도 하지만, 간단하게 설치까지는 쉽게 할 수 있으니깐 ㅜ

그리고 오픈소스 한두개만 정해서 제대로 공부해도 다른걸 쓸때도 어렵지않게 사용하거나 금방 배울 수 있는 것 같다.
https://airflow.apache.org/docs/apache-airflow/stable/start/local.html

데이터 파이프라인 블록을 묶거나 오케스트레이션하는 플랫폼 또는 프레임워크 (간단하게는 스케줄러 느낌)

Hadoop에 종속되지않음 Python으로 개발가능함

1. 환경 세팅 (리눅스 RedHat)

kerberos openldap과 같은 패키지들 선 설치필요

  sudo yum install -y \
        freetds \
        freetds-devel \
        krb5-server krb5-libs krb5-auth-dialog \
        openldap openldap-clients openldap-servers \
        libffi \
        cyrus-sasl-plain cyrus-sasl\ (libsasl2  대신 )
        openssl-devel \
        redhat-lsb \
        sqlite \
        unixODBC

근데 보안까지 설정하면서 공부하지않을 예정이라 위과정은 생략 추후를 위해 (공식사이트엔 우분투기준 레드햇계열 찾아보던가해야한다)

2. Airflow 다운로드 방법

1. Apache Mirror에서 다운 https://dlcdn.apache.org/airflow/2.2.4/apache-airflow-2.2.4.tar.gz
2. Github git clone -b v2-2-stable --single-branch https://github.com/apache/airflow.git
3. 공식사이트 https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-sources.html

1번 방법으로 Airflow 설치

$ sudo wget https://dlcdn.apache.org/airflow/2.2.4/apache-airflow-2.2.4.tar.gz --no-check-certificate

build

python3 -m venv /opt/apache-airflow-2.2.4/
source /opt/apache-airflow-2.2.4/bin/activate

airflow 실행

./airflow webserver -p 8080
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/