Airflow Access Variable From Previous Python Operator

Question

I'm new to Airflow and I'm currently building a DAG that will execute a PythonOperator, a BashOperator, and then another PythonOperator structured like this:

def authenticate_user(**kwargs):
    ...
    list_prev = [...]

AUTHENTICATE_USER = PythonOperator(
        task_id='AUTHENTICATE_USER',
        python_callable=authenticate_user,
        provide_context=True,
        dag=dag)

CHANGE_ROLE = BashOperator(
        task_id='CHANGE_ROLE',
        bash_command='...',
        dag=dag)

def calculations(**kwargs):
    list_prev
    ...

CALCULATIONS = PythonOperator(
    task_id='CALCULATIONS',
    python_callable=calculations,
    provide_context=True,
    dag=dag)

My issue is, I create a list of variables in the first PythonOperator (AUTHENTICATE_USER) that I would like to use later in my second PythonOperator (CALCULATIONS) after executing the BashOperator (CHANGE_ROLE). Is there a way for me to carry over that created list into other PythonOperators in my current DAG?

Thank you

y2k-shubham · Accepted Answer · 2019-08-15 18:22:15Z

2

I can think of 3 possible ways (to avoid confusion with the Airflow's concept of Variable, I'll call the data that you want to share between tasks as values)

Airflow XCOMs: Push your values from AUTHENTICATE_USER task and pull them in your CALCULATIONS task. You can either publish and access each value separately or wrap them all into a Python dict or list (better as it reduces db reads and writes)
External system: Persist your values from 1st task into some external system such as database, files or S3-objects and access them from downstream tasks when needed
Airflow Variables: This is a specific case of point (2) above (as Variables are stored in Airflow's backend meta-db). You can programmatically create, modify or delete Variables by exploiting the underlying SQLAlchemy model. See this for hints.

answered Aug 15, 2019 at 18:22

y2k-shubham

11.9k15 gold badges75 silver badges166 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Airflow Access Variable From Previous Python Operator

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related