Postgresql on AWS Lambda Ruby
amazon web services - Ruby gem with native extensions not working on AWS Lambda - Stack Overflow
ruby - How to correctly load a gem extension in AWS Lambda - Stack Overflow
Ruby 3.2 runtime now available in AWS Lambda
Ever since AWS announced that Ruby was now available on Lambdas I have been looking at porting over my RoR code to Lambdas to see if I can save money instead of running EC2 instances. Everything is going well and I have been able to almost successfully run all of my gems except 'pg' on the lambci/lambda:ruby2.5 docker image provided by the good folks at LambCI. I use this docker image to emulate locally what environment I would be given in AWS Lambda.
Has anyone tried to set up an Postgresql adapter on AWS Lambda Ruby?
Has anyone tried to do it on the LambCI Docker container?
I only receive the following
I am also far from a Docker expert
{
"errorType": "Init\u003cLoadError\u003e",
"errorMessage": "Error loading the 'postgresql' Active Record adapter. Missing a gem it depends on?
libruby.so.2.5: cannot open shared object file: No such file or directory -
/var/task/vendor/bundle/ruby/2.5.0/gems/pg-1.1.3/lib/pg_ext.so",
"stackTrace": [
"/var/lang/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'",Edit: I was finally able to get the lambci/lambda:ruby2.5 up and running by adding the libpq.so.5 to a directory that I created called lib. If you look at the https://github.com/lambci/docker-lambda/blob/master/ruby2.5/run/Dockerfile it has a line LD_LIBRARY_PATH. By default the $LD_LIBRARY_PATH includes /var/task/lib
ENV PATH=/var/lang/bin:$PATH \ LD_LIBRARY_PATH=/var/lang/lib:$LD_LIBRARY_PATH \ AWS_EXECUTION_ENV=AWS_Lambda_ruby2.5
The overall steps I took was to build the gems via the build-ruby docker image in my ruby directory
docker run -v "$PWD":/var/task -it lambci/lambda:build-ruby2.5 /bin/bash -c "yum -y install postgresql-devel postgresql-libs ; bundle install --deployment ; bash"
Find the library libpq.so.5 in the file system (/usr/lib64/ was my location) and create directory called lib in the ruby folder and copy the libpq file into it.
Finally run the docker again
docker run --rm -v "$PWD":/var/task -i lambci/lambda:ruby2.5 file.lambda_handler
Running ldd on gems/oj-2.18.5/lib/oj/oj.so clarifies the first error. the problem is not that oj.so does not exist, but that libruby.so.2.5 does not exist. The second problem is that the current Ruby lambda has glibc version 2.17, where AWS-linux comes with glibc version 2.25.
The fundamental problem here is that you need to install your gems on a system identical to the one they will run on if you have native dependencies. The best way I've found to do this is using docker. https://github.com/lambci/docker-lambda has docker images of lambda installs.
For ruby, do the following:
- Build the docker ruby image
- Copy your gemfile into a pristine directory
- From that directory, run:
docker run -v "$PWD":/var/task --entrypoint bundle lambci/lambda-base:ruby2.5 install --path=/var/task
This will give you a folder called ruby with a version dependencies compatible with lambda.
If you are planning to use this output with lambda layers, keep in mind that the file structure produced by bunlde is ruby/2.5.0/... and it needs need to be ruby/gems/2.5.0/... before you upload it.
The problem is precisely as you stated, to summarize it: native extension gems need to be built in the same environment as they will be run. Therefore, before uploading the vendor to aws, we must install the gems in an environment which is similar to that of lambda.
To build your vendor/bundle in a way compliant with lambda, use the following docker container:
docker run --rm -v "$PWD":/var/task lambci/lambda:build-ruby2.7 bundle install --deployment
This will give you a working vendor/bundle dependency.
You can run another container to check if the function works:
docker run --rm -v $PWD:/var/task:ro,delegated lambci/lambda:ruby2.7 lambda_handler.lambda_handler
The full list of images is here: https://github.com/lambci/docker-lambda
The question's already 1 year old but I had to search for about 2 hours to solve the exact same problem with the gem nokogiri, so I thought this might be useful to somebody.
Hope this spares you from some googling hours!
Do you have libpq.so.5 on your lib folder?
Your error is saying that did not find libpq.so.5 on the $PATH, in AWS Lambda the folder lib is automatically loaded on the path, thus, you just need to have this file there.
Executables created outside the lambda world do not run on Lambda, furthermore, you need to compile the executables by your own on a Lambda image. This is an example in how to do that:
Gemfile
source "https://rubygems.org"
gem "pg"
gem "mysql2"
handler.rb
require 'pg'
require 'mysql2'
def run(event:, context:)
{
postgres_client_version: PG.library_version,
mysql_client_version: Mysql2::VERSION
}
end
Dockerfile
FROM lambci/lambda:build-ruby2.5
RUN yum install -y postgresql postgresql-devel mysql mysql-devel
RUN gem update bundler
ADD Gemfile /var/task/Gemfile
ADD Gemfile.lock /var/task/Gemfile.lock
RUN bundle install --path /var/task/vendor/bundle --clean
This is going to build your image, then run it to generate the PG and MYSQL executables, then copy it to your lib folder.
build.sh
#!/bin/bash -x
set -e
rm -rf lib && rm -rf vendor && mkdir lib && mkdir vendor
docker build -t pg_mysql_layer -f Dockerfile .
CONTAINER=$(docker run -d pg_mysql_layer false)
docker cp \
$CONTAINER:/var/task/vendor/ \
./
docker cp \
$CONTAINER:/usr/lib64/libpq.so.5.5 \
lib/libpq.so.5
docker cp \
$CONTAINER:/usr/lib64/mysql/. \
lib/
docker rm $CONTAINER
After running ./build.sh it is going to generate the folder lib and vendor with all you need, now you just need to deploy your lambda function.
To test locally you can run:
docker run --rm -it -v $PWD:/var/task -w /var/task lambci/lambda:ruby2.5 handler.run
It is going to return something similar to this:

REF: https://www.stevenringo.com/ruby-in-aws-lambda-with-postgresql-nokogiri/
REF: https://www.reddit.com/r/ruby/comments/a3e7a1/postgresql_on_aws_lambda_ruby/
There are few good plugins to manage dependencies for AWS lambda. serverless-ruby-layer for ruby and serverless-python-requirements for python.
For your ruby case, you can use serverless-ruby-layer simply by adding plugin related config to your serverless.yml.
service: using-docker-yums
plugins:
- serverless-ruby-layer
custom:
rubyLayer:
use_docker: true
docker_yums:
- postgresql-devel
native_libs:
- /usr/lib64/libpq.so.5
provider:
name: aws
runtime: ruby2.5
functions:
hello:
handler: handler.hello
And you need to install the plugin with the below command inside your serverless project folder,
sls plugin install -n serverless-ruby-layer
Now running sls deploy will automatically deploy the gems and libs to the layer.
Check out this example here in the docs