I've seen that a few libraries like numpy and pandas don't work in Lambda when installed using pip. I have had success using the .whl package files for these libraries to create the Lambda layer. Refer to the steps below:

NOTE: These steps set up the libraries specific to the Python 3.7 runtime. If using any other version, you would need to download the .whl files corresponding to that Python version.

1. Create an EC2 instance using Amazon Linux AMI and SSH into this instance. We should create our layer in Amazon Linux AMI as the Lambda Python 3.7 runtime runs on this operating system (doc).

2. Make sure this instance has Python3 and "pip" tool installed.

3. Download the numpy .whl file for the cp37 Python version and the manylinux1_x86_64 OS by executing the below command:

$ wget https://files.pythonhosted.org/packages/d6/c6/58e517e8b1fb192725cfa23c01c2e60e4e6699314ee9684a1c5f5c9b27e1/numpy-1.18.5-cp37-cp37m-manylinux1_x86_64.whl

4. Skip to the next step if you're not using pandas. Download the pandas .whl file for the cp37 Python version and the manylinux1_x86_64 OS by executing the below command:

$ wget https://files.pythonhosted.org/packages/a4/5f/1b6e0efab4bfb738478919d40b0e3e1a06e3d9996da45eb62a77e9a090d9/pandas-1.0.4-cp37-cp37m-manylinux1_x86_64.whl

5. Next, we will create a directory named "python" and unzip these files into that directory:

        $ mkdir python
        $ unzip pandas-1.0.4-cp37-cp37m-manylinux1_x86_64.whl -d python/
        $ unzip numpy-1.18.5-cp37-cp37m-manylinux1_x86_64.whl -d python/

6. We also need to download "pytz" library to successfully import numpy and pandas libraries:

        $ pip3 install -t python/ pytz

7. Next, we would remove the “*.dist-info” files from our package directory to reduce the size of the resulting layer.

        $ cd python
        $ sudo rm -rf *.dist-info

8. This will install all the required libraries that we need to run pandas and numpy.

9. Zip the current "python" directory and upload it to your S3 bucket. Ensure that the libraries are present in the hierarchy as given here.

        $ cd ..
        $ zip -r lambda-layer.zip python/
        $ aws s3 cp lambda-layer.zip s3://YOURBUCKETNAME

10. The "lambda-layer.zip" file can then be used to create a new layer from the Lambda console.

If I may, I would like to recommend an alternative technique which has never failed me. The technique includes docker tool described in the recent AWS blog:

• How do I create a Lambda layer using a simulated Lambda environment with Docker?

Thus for this question, I verified it using elasticsearch as follows:

1. Create empty folder, e.g. mylayer.

2. Go to the folder and create requirements.txt file with the content of

elasticsearch

3. Run the following docker command (may adjust python version to your needs):

docker run -v "$PWD":/var/task "lambci/lambda:build-python3.8" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit"

4. Create layer as zip:

zip -r elastic.zip python > /dev/null

5. Create lambda layer based on elastic.zip in the AWS Console. Don't forget to specify Compatible runtimes to python3.8.

6. Test the layer in lambda using the following lambda function:

import json

from elasticsearch import Elasticsearch, RequestsHttpConnection

def lambda_handler(event, context):
    # TODO implement
    
    print(dir(Elasticsearch))
    
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

The function executes correctly:

['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__enter__', '__eq__', '__exit__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'bulk', 'clear_scroll', 'close', 'count', 'create', 'delete', 'delete_by_query', 'delete_by_query_rethrottle', 'delete_script', 'exists', 'exists_source', 'explain', 'field_caps', 'get', 'get_script', 'get_script_context', 'get_script_languages', 'get_source', 'index', 'info', 'mget', 'msearch', 'msearch_template', 'mtermvectors', 'ping', 'put_script', 'rank_eval', 'reindex', 'reindex_rethrottle', 'render_search_template', 'scripts_painless_execute', 'scroll', 'search', 'search_shards', 'search_template', 'termvectors', 'update', 'update_by_query', 'update_by_query_rethrottle']