Run python script inside docker inside Airflow

Question

I have this code for airflow to execute python script inside docker :

a02_retrieve_2 = SSHExecuteOperator(
    task_id='a02_retrieve_2',
    ssh_hook=sshHook,
    dag=dag,
    bash_command= 'docker exec myDocker -i bash -c "  /opt/conda/bin/python aretrieve_2.py   --myarg  45  --myarg 35   "  '
)

Unfortunately, it does not work.

But, the version with no argument works.

a02_retrieve_2 = SSHExecuteOperator(
    task_id='a02_retrieve_2',
    ssh_hook=sshHook,
    dag=dag,
    bash_command= 'docker exec myDocker -i bash -c "  /opt/conda/bin/python aretrieve_2.py  "  '
)

Error is :

Python script starts to run but Airflow is unable to catch up the python script parameters...

How to make the version with python script arguments works ?

Is this related to jinja template ? BashOperator doen't run bash file apache airflow

You already asked this question and probably didn't get an answer because you haven't provided enough information. What does it say in your logs. — Simon D
– Simon D, Commented Apr 26, 2018 at 9:04
Does everything work fine if you execute bash -c " /opt/conda/bin/python aretrieve_2.py --myarg 45 --myarg 35 " directly on the machine? — tobi6
– tobi6, Commented May 2, 2018 at 7:28
yes, of course. this is related to jinja template in airflow — tensor
– tensor, Commented May 2, 2018 at 13:57

tensor · Accepted Answer · 2018-05-06 14:45:16Z

0

I found the answer by looking at the source code of bashoperator: This uses popen

            self.log.info("Running command: %s", bash_command)
            sp = Popen(
                ['bash', fname],
                stdout=PIPE, stderr=STDOUT,
                cwd=tmp_dir, env=self.env,
                preexec_fn=pre_exec)

https://airflow.incubator.apache.org/_modules/bash_operator.html

answered May 6, 2018 at 14:45

tensor

3,39010 gold badges43 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Run python script inside docker inside Airflow

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related