Numpy Import Error and Aws lambda layer folder structure (Importing the numpy C-extensions failed)

Problem

Hi,

I've been trying to create two lambda layers containing numpy(1.23.2) and pandas(1.4.4) packages respectively which I would then use with my lambda function. (note: I'm also inserting in the pandas layer pytz-2022.2.1)

However, when I test it through the UI I get the following error:

[ERROR] ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.9 from "/var/lang/bin/python3.9"
  * The NumPy version is: "1.23.2"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

I have been following aws guides, articles and other articles in stack overflow without success.

Steps

I am getting the numpy package using this link from pypi numpy-1.23.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

I unzip the .whl and zip the content into a new .zip folder. The structure of this new zip is as follows:

numpy-layer.zip
└─python
     └─numpy
     └─numpy-1.23.2.dist-info
     └─numpy.libs

(just to be clear, when I extract the files I get a folder called python)

I push the zip to an s3 bucket:

aws s3 cp  $dir/lambda-numpy-layer.zip s3://my-bucket/lambda_packages/numpy.zip

I publish a new layer version:

aws lambda publish-layer-version \
  --layer-name my-numpy-layer  \
  --description "Using numpy 1.23.2"  \
  --content S3Bucket=my-bucket,S3Key=lambda_packages/numpy.zip \
  --output json

and update the lambda function configuration to use the new layers:

aws lambda update-function-configuration \
  --function-name my-function-name \
  --layers  $numpy_layer_arn_with_version \
            $pandas_layer_arn_with_version

(note, I checked if it was an issue with the aws cli commands so I tried uploading the zip through the UI into the lambda layer but the error is still the same)

Debug

When trying to debug, I printed out some folder structures.

def lambda_handler(event, context):
    
    import os
    print("Current dir", os.getcwd())
    print("listdit /root", os.listdir("/"))
    print("listdir /opt", os.listdir("/opt"))
    print("listdir /opt/python", os.listdir("/opt/python"))
    print("listdir /var/lang/bin/", os.listdir('/var/lang/bin/'))
    print("PYTHONPATH:", os.environ.get('PYTHONPATH'))
    print("PATH:", os.environ.get('PATH'))
    import numpy

    return {
        'headers': {'Content-Type': 'application/json'}
        , 'statusCode': 200
        , 'body': body
    }

Which outputs:

Current dir /var/task
listdit /root ['proc', 'media', 'sys', 'sbin', 'lib64', 'tmp', 'opt', 'dev', 'mnt', 'root', 'etc', 'home', 'bin', 'usr', 'var', 'run', 'boot', 'lambda-entrypoint.sh', 'lib', 'srv', 'THIRD-PARTY-LICENSES.txt']
listdir /opt ['python']
listdir /opt/python ['numpy', 'numpy-1.23.2.dist-info', 'numpy.libs', 'pandas', 'pandas-1.4.4.dist-info', 'pandas.libs', 'pytz', 'pytz-2022.2.1.dist-info']
listdir /var/lang/bin/ ['pydoc3.9', 'pydoc3', 'python3.9-config', 'python-config', 'python3.9', 'pip', 'python3-config', 'pip3.9', 'pydoc', 'pip3', 'python', 'python3']
PYTHONPATH: /var/runtime
PATH: /var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin

[ERROR] ImportError: 

...

Could be that the path is pointing at /opt/bin but my packages are directly under /opt/ instead? Otherwise a problem with my zipped folder structure.