How to pass parameter to PythonOperator in Airflow

Question

I just started using Airflow, can anyone enlighten me how to pass a parameter into PythonOperator like below:

t5_send_notification = PythonOperator(
    task_id='t5_send_notification',
    provide_context=True,
    python_callable=SendEmail,
    op_kwargs=None,
    #op_kwargs=(key1='value1', key2='value2'),
    dag=dag,
)

def SendEmail(**kwargs):
    msg = MIMEText("The pipeline for client1 is completed, please check.")
    msg['Subject'] = "xxxx"
    msg['From'] = "xxxx"
    ......
    s = smtplib.SMTP('localhost')
    s.send_message(msg)
    s.quit()

I would like to be able to pass some parameters into the t5_send_notification's callable which is SendEmail, ideally I want to attach the full log and/or part of the log (which is essentially from the kwargs) to the email to be sent out, guessing the t5_send_notification is the place to gather those information.

Thank you very much.

Ryan Yuan · Accepted Answer · 2019-02-26 23:27:45Z

71

Pass a dict object to op_kwargs

Use the keys to access their value from kwargs dict in your python callable

def SendEmail(**kwargs):
    print(kwargs['key1'])
    print(kwargs['key2'])
    msg = MIMEText("The pipeline for client1 is completed, please check.")
    msg['Subject'] = "xxxx"
    msg['From'] = "xxxx"
    ......
    s = smtplib.SMTP('localhost')
    s.send_message(msg)
    s.quit()


t5_send_notification = PythonOperator(
    task_id='t5_send_notification',
    provide_context=True,
    python_callable=SendEmail,
    op_kwargs={'key1': 'value1', 'key2': 'value2'},
    dag=dag,
)

answered Feb 26, 2019 at 23:27

Ryan Yuan

2,5762 gold badges15 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Amin Over a year ago

that didn't work. it should be like: SendEmail(key1, key2, **kwargs):

Ryan Yuan Over a year ago

@Amin which version of the airflow you are using? the reason why I asked is that you might be using python3 as the latest versions of airflow support python3 much better than a year ago, but still there are lots of people using python2 for airflow dev.

Amin Over a year ago

If it has something to do with python version please mention it in the answer.

jahmia · Accepted Answer · 2020-09-10 23:37:55Z

24

PythonOperator have a named parameter op_kwargs and accepts dict object.

have

t5_send_notification = PythonOperator(
    task_id='t5_send_notification',
    provide_context=True,
    python_callable=SendEmail,
    op_kwargs={"my_param":'value1'},
    dag=dag,
)

def SendEmail(my_param,**kwargs):
    print(my_param) #'value_1'
    msg = MIMEText("The pipeline for client1 is completed, please check.")
    msg['Subject'] = "xxxx"
    msg['From'] = "xxxx"
    ......
    s = smtplib.SMTP('localhost')
    s.send_me

edited Sep 10, 2020 at 23:37

jahmia

651 silver badge10 bronze badges

answered Feb 26, 2019 at 21:42

ethanenglish

1,3572 gold badges16 silver badges34 bronze badges

3 Comments

mdivk Over a year ago

Thank you Ethan, your code of op_kwargs={my_param='value1'}, reports error.

Joseph Seung Jae Dollar Over a year ago

{key : value}

ethanenglish Over a year ago

Do you have the entire error? Is the key in quotes as well?

Collectives™ on Stack Overflow

How to pass parameter to PythonOperator in Airflow

2 Answers 2

3 Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related