AWS lambda doesn't recognize python modules in deployment package

Question

When I upload a (zipped) deployment package as a lambda function on AWS I get "no module named..." errors for both bs4 and google.

I created a virtual environment using venv and I installed the required dependencies The app works fine when running from within the virtual environment. But, when I zip it up and upload it as a lambda function on AWS, I get "no module named..." errors for both "bs4" and (if I hash out the import of bs4 for debug reasons) also for "google". I checked the site-packages folder in the zip file and they seem to be there.

Why is AWS saying there is no module when there is?!

I am using python3.6 on Ubuntu.

This seems interesting. I had a makefile that would package all the content in my venv and upload it to Lambda. Can you share a screen/directory listing of your ZIP Folder? — Rhythem Aggarwal
– Rhythem Aggarwal, Commented Jul 31, 2018 at 14:05
It is a very long tree and I am struggling to export it to anything other than the terminal. What to do ? :-( Could I send you the zip file? — Gingmeister
– Gingmeister, Commented Jul 31, 2018 at 20:03
Oke let me screenshot you the absolute basics u would need in ur ZIP for ur lambda to be able to recognize everything properly — Rhythem Aggarwal
– Rhythem Aggarwal, Commented Aug 1, 2018 at 6:46
imgur.com/a/6FZk3Qm Here is a screen of what my working ZIP Looks like. 1 is the list of installed python modules. 2 is my main python code. Everything else is default python libraries. — Rhythem Aggarwal
– Rhythem Aggarwal, Commented Aug 1, 2018 at 6:51

Rhythem Aggarwal · Accepted Answer · 2018-08-01 06:59:26Z

1

Lambda needs ZIP with all the libraries and your main python code file in the same folder. Here is what i do:

Create a new package folder with the following hierarchy

mkdir -p ./package/tmp/lib
Copy Project folder into the temp folder

cp -a ./$(PROJECT)/. ./package/tmp/
Copy python site-packages from virtual env to temp folder in package

cp -a $(VIRTUAL_ENV)/lib/python2.7/site-packages/. ./package/tmp/
Remove any unused libraries (that are not required for this particular lambda to run) from temp folder
```
rm -rf ./package/tmp/wheel*
```
Zip the temp package directory

cd ./package/tmp && zip -r ../$(PROJECT).zip .

This final zip so created is ready for upload on Lambda.

answered Aug 1, 2018 at 6:59

Rhythem Aggarwal

3562 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Gingmeister Over a year ago

That's great. Thank you very much. I'll try that later today.

Gingmeister Over a year ago

Update - I havent yet got the Lambda running successfully as I have encountered another problem - but it seems to be as simple as moving all the site packages to the same folder as the scripts and zipping up.

Rhythem Aggarwal Over a year ago

yes, ideally all site packages should be in ur script library. What issue are u facing now?

Gingmeister Over a year ago

One of the python modules I am using (googlesearch) wants to write to disk and AWS Lambda only allows writing to /tmp.

Rhythem Aggarwal Over a year ago

Oh yes, that is a completely different problem. I would suggest you close this question, and ask a seperate question on it. As a hint, i would suggest you look into "S3" storage provided by AWs. i think it would solve ur problem perfectly. Make a new question and if i see i'll respond to that. As for the scope of this one, i feel the problem of Lambda not recognizing ur python libraries is solved.

|

Collectives™ on Stack Overflow

AWS lambda doesn't recognize python modules in deployment package

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related