Airflow how to import python script?

Question

I have a file survey_monkey.py and a directory surveymonkey in the dag folder. There is a python script named surveys.py in the directory surveymonkey.

from surveys import Surveys

# defining DAG arguments

# You can override them on a per-task basis during operator initialization
default_args = {
    'owner': 'survey_monkey'，
    'start_date': days_ago(0),
    'email': ['[email protected]'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

This givs me an error saying

Broken DAG: [/home/ubuntu/airflow/dags/survey_tools/survey_monkey.py] Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ubuntu/airflow/dags/survey_tools/survey_monkey.py", line 11, in <module>
    from surveys import Surveys
ModuleNotFoundError: No module named 'surveys'

How can I fix it?

Nitin Bhojwani · Accepted Answer · 2022-03-13 13:59:07Z

1

The import path should be relative to dags folder. Since surveys stays inside surveymonkey which is in dag folder.

Replace following line:

from surveys import Surveys

with:

from surveymonkey.surveys import Surveys

answered Mar 13, 2022 at 13:59

Nitin Bhojwani

7221 gold badge5 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Elad Kalif · Accepted Answer · 2022-03-13 16:45:09Z

0

You can do one of the following:

add your modules to one of the folders that Airflow automatically adds to PYTHONPATH
add extra folders where you keep your code to PYTHONPATH
package your code into a Python package and install it together with Airflow.

Airflow has Modules Management documentation that explains it thoroughly.

In your case the module you wish to import exist in the DAG folder (Which already exist in PYTHONPATH) so you can simply import it with a path relative to your DAG folder. The following import should work:

from survey_tools.survey_monkey.surveys import Surveys

edited Mar 13, 2022 at 16:45

answered Mar 13, 2022 at 10:50

Elad Kalif

16.4k3 gold badges29 silver badges63 bronze badges

Collectives™ on Stack Overflow

Airflow how to import python script?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related