Postgresql on AWS Lambda Ruby
ruby - How to correctly load a gem extension in AWS Lambda - Stack Overflow
amazon web services - AWS Lambda Unable to Find Ruby Dependency - Stack Overflow
Ruby 3.2 runtime now available in AWS Lambda
Ever since AWS announced that Ruby was now available on Lambdas I have been looking at porting over my RoR code to Lambdas to see if I can save money instead of running EC2 instances. Everything is going well and I have been able to almost successfully run all of my gems except 'pg' on the lambci/lambda:ruby2.5 docker image provided by the good folks at LambCI. I use this docker image to emulate locally what environment I would be given in AWS Lambda.
Has anyone tried to set up an Postgresql adapter on AWS Lambda Ruby?
Has anyone tried to do it on the LambCI Docker container?
I only receive the following
I am also far from a Docker expert
{
"errorType": "Init\u003cLoadError\u003e",
"errorMessage": "Error loading the 'postgresql' Active Record adapter. Missing a gem it depends on?
libruby.so.2.5: cannot open shared object file: No such file or directory -
/var/task/vendor/bundle/ruby/2.5.0/gems/pg-1.1.3/lib/pg_ext.so",
"stackTrace": [
"/var/lang/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'",Edit: I was finally able to get the lambci/lambda:ruby2.5 up and running by adding the libpq.so.5 to a directory that I created called lib. If you look at the https://github.com/lambci/docker-lambda/blob/master/ruby2.5/run/Dockerfile it has a line LD_LIBRARY_PATH. By default the $LD_LIBRARY_PATH includes /var/task/lib
ENV PATH=/var/lang/bin:$PATH \ LD_LIBRARY_PATH=/var/lang/lib:$LD_LIBRARY_PATH \ AWS_EXECUTION_ENV=AWS_Lambda_ruby2.5
The overall steps I took was to build the gems via the build-ruby docker image in my ruby directory
docker run -v "$PWD":/var/task -it lambci/lambda:build-ruby2.5 /bin/bash -c "yum -y install postgresql-devel postgresql-libs ; bundle install --deployment ; bash"
Find the library libpq.so.5 in the file system (/usr/lib64/ was my location) and create directory called lib in the ruby folder and copy the libpq file into it.
Finally run the docker again
docker run --rm -v "$PWD":/var/task -i lambci/lambda:ruby2.5 file.lambda_handler
Do you have libpq.so.5 on your lib folder?
Your error is saying that did not find libpq.so.5 on the $PATH, in AWS Lambda the folder lib is automatically loaded on the path, thus, you just need to have this file there.
Executables created outside the lambda world do not run on Lambda, furthermore, you need to compile the executables by your own on a Lambda image. This is an example in how to do that:
Gemfile
source "https://rubygems.org"
gem "pg"
gem "mysql2"
handler.rb
require 'pg'
require 'mysql2'
def run(event:, context:)
{
postgres_client_version: PG.library_version,
mysql_client_version: Mysql2::VERSION
}
end
Dockerfile
FROM lambci/lambda:build-ruby2.5
RUN yum install -y postgresql postgresql-devel mysql mysql-devel
RUN gem update bundler
ADD Gemfile /var/task/Gemfile
ADD Gemfile.lock /var/task/Gemfile.lock
RUN bundle install --path /var/task/vendor/bundle --clean
This is going to build your image, then run it to generate the PG and MYSQL executables, then copy it to your lib folder.
build.sh
#!/bin/bash -x
set -e
rm -rf lib && rm -rf vendor && mkdir lib && mkdir vendor
docker build -t pg_mysql_layer -f Dockerfile .
CONTAINER=$(docker run -d pg_mysql_layer false)
docker cp \
$CONTAINER:/var/task/vendor/ \
./
docker cp \
$CONTAINER:/usr/lib64/libpq.so.5.5 \
lib/libpq.so.5
docker cp \
$CONTAINER:/usr/lib64/mysql/. \
lib/
docker rm $CONTAINER
After running ./build.sh it is going to generate the folder lib and vendor with all you need, now you just need to deploy your lambda function.
To test locally you can run:
docker run --rm -it -v $PWD:/var/task -w /var/task lambci/lambda:ruby2.5 handler.run
It is going to return something similar to this:
REF: https://www.stevenringo.com/ruby-in-aws-lambda-with-postgresql-nokogiri/
REF: https://www.reddit.com/r/ruby/comments/a3e7a1/postgresql_on_aws_lambda_ruby/
There are few good plugins to manage dependencies for AWS lambda. serverless-ruby-layer for ruby and serverless-python-requirements for python.
For your ruby case, you can use serverless-ruby-layer simply by adding plugin related config to your serverless.yml.
service: using-docker-yums
plugins:
- serverless-ruby-layer
custom:
rubyLayer:
use_docker: true
docker_yums:
- postgresql-devel
native_libs:
- /usr/lib64/libpq.so.5
provider:
name: aws
runtime: ruby2.5
functions:
hello:
handler: handler.hello
And you need to install the plugin with the below command inside your serverless project folder,
sls plugin install -n serverless-ruby-layer
Now running sls deploy will automatically deploy the gems and libs to the layer.
Check out this example here in the docs
The issue was a mismatch between Ruby versions. I was a few versions behind AWS Lambda's version (2.5.0). Once I changed updated my local version to the one Labmda uses, the loading error went away.
This issue is a duplicate of AWS Lambda: Ruby function failing to load gem and How can I get my AWS Lambda to access gems stored in vendor/bundle? but at this time, neither has accepted answers, so I'm leaving this posted here for future devs in a similar situation.
You do not need to match your local Ruby version to the AWS Ruby version. Instead you can utilise Docker to vendor the 2.5.0 gems in this way:
cd /path/to/Gemfile/ && \
docker run -v `pwd`:`pwd` -w `pwd` -i -t lambci/lambda:build-ruby2.5 bundle install --deployment