Pip install Python package within AWS Lambda?

Question

I'm trying to pip install a package in an AWS Lambda function.

The method recommended by Amazon is to create a zipped deployment package that includes the dependencies and python function all together (as described in AWS Lambda Deployment Package in Python). However, this results in not being able to edit the Lambda function using inline code editing within the AWS Lambda GUI.

So instead, I would like to pip install the package within the AWS Lambda function itself. In AWS Lambda, the filesystem is read-only apart from the /tmp/ directory, so I am trying to pip install to the /tmp/ directory. The function is only called once-daily, so I don't mind about the few extra seconds required to re-pip install the package every time the function is run.

My attempt

def lambda_handler(event, context):
    # pip install dependencies
    print('begin lambda handler')
    import subprocess
    import sys
    subprocess.call('pip install cryptography -t /tmp/ --no-cache-dir'.split())
    from cryptography.fernet import Fernet
    pwd_encrypted = b'gAAAAABeTcT0OXH96ib7TD5-sTII6jMfUXPhMpwWRCF0315rWp4C0yav1XAPIn7prfkkA4tltYiWFAJ22bwuaj0z1CKaGl8vTgNd695SDl25HnLwu1xTzaQ='
    key = b'fP-7YR1hUeVW4KmFmly4JdgotD6qjR52g11RQms6Llo='
    cipher_suite = Fernet(key)
    result = cipher_suite.decrypt(pwd_encrypted).decode('utf-8')
    print(result)
    print('end lambda handler')

However, this results in the error

[ERROR] ModuleNotFoundError: No module named 'cryptography'

I have also tried replacing the subprocess call with the following, as recommended in this stackoverflow answer

    cmd = sys.executable+' -m pip install cryptography -t dependencies --no-cache-dir'
    subprocess.check_call(cmd.split())

However, this results in the error

OSError: [Errno 30] Read-only file system: '/var/task/dependencies'

I would not recommend running pip inside Lambda. You could package your dependencies into an AWS Lambda Layer, which would then let you edit the remaining code in the Lambda management console. — John Rotenstein
– John Rotenstein, Commented Feb 20, 2020 at 2:55
@JohnRotenstein - Why is this not recommended? I'm not intending to use this in production. It would make the debugging/learning process easier if everything can be self-contained within a single python file. Are there any blockers to making this work? — maurera
– maurera, Commented Feb 20, 2020 at 17:17
Yes, the blockers are exactly what you are experiencing. An AWS Lambda function can only write to the /tmp/ directory. — John Rotenstein
– John Rotenstein, Commented Feb 20, 2020 at 20:31
@JohnRotenstein - I realize that, and that's why I'm attempting to only write to the /tmp/ directory by specifying /tmp/ as the install target. Is there a way to then import the package, once the package is pip installed to /tmp/? The two methods I've tried of pip-installing to a custom directory seem to work outside of AWS Lambda — maurera
– maurera, Commented Feb 20, 2020 at 21:27

maurera · Accepted Answer · 2020-02-20 23:01:04Z

35

I solved this with a one-line adjustment to the original attempt. You just need to add /tmp/ to sys.path so that Python knows to search /tmp/ for the module. All you need to do is add the line sys.path.insert(1, '/tmp/').

Solution

import os
import sys
import subprocess

# pip install custom package to /tmp/ and add to path
subprocess.call('pip install cryptography -t /tmp/ --no-cache-dir'.split(), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
sys.path.insert(1, '/tmp/')
from cryptography.fernet import Fernet

def lambda_handler(event, context):
    # pip install dependencies
    pwd_encrypted = b'gAAAAABeTcT0OXH96ib7TD5-sTII6jMfUXPhMpwWRCF0315rWp4C0yav1XAPIn7prfkkA4tltYiWFAJ22bwuaj0z1CKaGl8vTgNd695SDl25HnLwu1xTzaQ='
    key = b'fP-7YR1hUeVW4KmFmly4JdgotD6qjR52g11RQms6Llo='
    cipher_suite = Fernet(key)
    result = cipher_suite.decrypt(pwd_encrypted).decode('utf-8')
    print(result)

Output

Hello stackoverflow!

Note - as @JohnRotenstein mentioned in the comments, the preferred method to add Python packages is to package dependencies in an AWS Lambda Layer. My solution just shows that it is possible to pip install packages directly in an AWS Lambda function.

answered Feb 20, 2020 at 23:01

maurera

1,6893 gold badges18 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Zenul_Abidin Over a year ago

It's not the best solution, because pip installation processes take several seconds to finish which adds up to your bill.

maurera Over a year ago

@Zenul_Abidin - you're right. As written in the answer, the preferred method to include a python package is to use an AWS Lambda Layer. However, the question asks whether it's possible to do this within an the Lambda function itself (eg - for testing purposes)

aykcandem Over a year ago

Whenever your lambda function is invoked, then it will try to install library which slows your code down. creating a package with all required libraries is better approach if you have time constraint. (all simply if you dont want to pay much) Check out deployment package with dependencies section in official documentation docs.aws.amazon.com/lambda/latest/dg/python-package.html

jameslol Over a year ago

I love this hack solution. I hope to never use it, but it's awesome. If your total layer bundle creeps over 250MB, this could help out in a pinch, but then you should obv shift over to container-back Lambdas. In response to the other commenters, no, this won't execute every time your function is invoked. Code outside of the function is only run when the function is initialised, which is only the first time it is invoked, i.e., cold start

maurera Over a year ago

@jameslol - thanks for that clarification! I hadn't realized that that code only gets executed during a cold start. I just googled about it and also found this reference, which seems to say that cold starts also happen if it's been ~5-7 minutes since the last invocation - mikhail.io/serverless/coldstarts/aws

|

Oliver Paul K · Accepted Answer · 2024-04-18 06:08:21Z

You can use any python package you need in a lambda using a lambda layers.

Have desired version of Python installed

❯ python3 --version
Python 3.12.2

Create a folder for the package and cd into it

mkdir py_packages
cd py_packages

Create a virtual Env

python3 -m venv venv
source venv/bin/activate

Create a python folder. This is the folder which is going to have all the files.

mkdir python
cd python

Install Requests package. You can install any package of your choice.

pip install requests -t .

Let's save space and delete objects with the ".dis-info" and the "venv" folders. They are not needed.

rm -rf *dist-info
rm -rf venv

Zip the folder to upload to aws lambda layer

cd ..
zip -r requets-package.zip python

Upload to S3 to directly upload via the lambda layer console if it's a small package

aws s3 cp requets-package.zip s3://s3-bucket-name/

Now you can create a new lambda layer or new version and use this zip file URL or upload from your local system. And attach to a lambda to access the package.

Laurie · Accepted Answer · 2020-10-08 11:17:15Z

1

For some reason subprocess.call() was returning a FileNotFound error when I was trying to pip3.8 install <package> -t <install-directory>. I solved this by using os.system() instead of subprocess.call(), and I specified the path of pip directly:

os.system('/var/lang/bin/pip3.8 install <package> -t <install-directory>').

answered Oct 8, 2020 at 11:17

Laurie

1,2171 gold badge13 silver badges29 bronze badges

Collectives™ on Stack Overflow

Pip install Python package within AWS Lambda?

3 Answers 3

7 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

7 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related