🌐
GitHub
github.com › lambci › lambci
GitHub - lambci/lambci: A continuous integration system built on AWS Lambda
LambCI includes a script you can source before running your build commands that will install Ruby, rbenv, gem and bundler. Call it with the Ruby version you want (currently: 2.7.0, 2.6.5, 2.5.7, 2.4.9, 2.3.8, 2.2.10, 2.1.10 or 2.0.0-p648): { ...
Starred by 4K users
Forked by 191 users
Languages   Shell 54.6% | JavaScript 42.8% | Dockerfile 1.9% | Ruby 0.7% | Shell 54.6% | JavaScript 42.8% | Dockerfile 1.9% | Ruby 0.7%
🌐
Medium
medium.com › @joshua.a.kahn › exploring-aws-lambda-layers-and-ruby-support-5510f81b4d14
Exploring AWS Lambda Layers and Ruby Support | by Josh Kahn | Medium
February 1, 2019 - In order to support Gems with native extensions, we can use Docker to replicate the Lambda environment locally and build there. The following is my approach to building and packaging a layer. This approach assumes that you have Docker installed, all dependencies are listed in a Gemfile, and all Ruby (.rb) files are to bundled as shared code: $ mkdir ruby && mkdir ruby/gems $ docker run --rm \ -v $PWD:/var/layer \ -w /var/layer \ lambci/lambda:build-ruby2.5 \ bundle install --path=ruby/gems# move directories and throw out cache $ mv ruby/gems/ruby/* ruby/gems/ && \ rm -rf ruby/gems/2.5.0/cache && \ rm -rf ruby/gems/ruby# bundle shared code $ mkdir ruby/lib && \ cp *.rb ruby/lib# zip and clean-up $ zip -r layer.zip ruby $ rm -rf .bundle && rm -rf ruby
Discussions

Postgresql on AWS Lambda Ruby
Probably that the pg libs are not on the Lambda server: Requirements PostgreSQL 9.0 or later (with headers, -dev packages, etc). https://bitbucket.org/ged/ruby-pg/wiki/Home More on reddit.com
🌐 r/ruby
34
12
December 7, 2018
amazon web services - Ruby gem with native extensions not working on AWS Lambda - Stack Overflow
The second problem is that the current Ruby lambda has glibc version 2.17, where AWS-linux comes with glibc version 2.25. The fundamental problem here is that you need to install your gems on a system identical to the one they will run on if you have native dependencies. The best way I've found to do this is using docker. https://github.com/lambci... More on stackoverflow.com
🌐 stackoverflow.com
ruby - How to correctly load a gem extension in AWS Lambda - Stack Overflow
After running ./build.sh it is going to generate the folder lib and vendor with all you need, now you just need to deploy your lambda function. To test locally you can run: docker run --rm -it -v $PWD:/var/task -w /var/task lambci/lambda:ruby2.5 handler.run More on stackoverflow.com
🌐 stackoverflow.com
Ruby 3.2 runtime now available in AWS Lambda
Glad to have it though frustrating that I'll have to jump straight from 2.7 to 3.2. More on reddit.com
🌐 r/ruby
7
58
June 9, 2023
🌐
Hint
hint.io › blog › lambda-layer-dependencies
Ruby on AWS Lambda: Layer Dependencies
July 13, 2020 - # Use AWS Lambda ruby2.7 build environment FROM lambci/lambda:build-ruby2.7
🌐
Reddit
reddit.com › r/ruby › postgresql on aws lambda ruby
r/ruby on Reddit: Postgresql on AWS Lambda Ruby
December 7, 2018 -

Ever since AWS announced that Ruby was now available on Lambdas I have been looking at porting over my RoR code to Lambdas to see if I can save money instead of running EC2 instances. Everything is going well and I have been able to almost successfully run all of my gems except 'pg' on the lambci/lambda:ruby2.5 docker image provided by the good folks at LambCI. I use this docker image to emulate locally what environment I would be given in AWS Lambda.

Has anyone tried to set up an Postgresql adapter on AWS Lambda Ruby?

Has anyone tried to do it on the LambCI Docker container?

I only receive the following

I am also far from a Docker expert

{
"errorType": "Init\u003cLoadError\u003e",
"errorMessage": "Error loading the 'postgresql' Active Record adapter. Missing a gem it depends on? 
libruby.so.2.5: cannot open shared object file: No such file or directory - 
/var/task/vendor/bundle/ruby/2.5.0/gems/pg-1.1.3/lib/pg_ext.so",
"stackTrace": [
"/var/lang/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'",

Edit: I was finally able to get the lambci/lambda:ruby2.5 up and running by adding the libpq.so.5 to a directory that I created called lib. If you look at the https://github.com/lambci/docker-lambda/blob/master/ruby2.5/run/Dockerfile it has a line LD_LIBRARY_PATH. By default the $LD_LIBRARY_PATH includes /var/task/lib

ENV PATH=/var/lang/bin:$PATH \
LD_LIBRARY_PATH=/var/lang/lib:$LD_LIBRARY_PATH \
AWS_EXECUTION_ENV=AWS_Lambda_ruby2.5
  1. The overall steps I took was to build the gems via the build-ruby docker image in my ruby directory

    docker run -v "$PWD":/var/task -it lambci/lambda:build-ruby2.5 /bin/bash -c "yum -y install postgresql-devel postgresql-libs ; bundle install --deployment ; bash"

  2. Find the library libpq.so.5 in the file system (/usr/lib64/ was my location) and create directory called lib in the ruby folder and copy the libpq file into it.

  3. Finally run the docker again

    docker run --rm -v "$PWD":/var/task -i lambci/lambda:ruby2.5 file.lambda_handler

🌐
Stevenringo
stevenringo.com › ruby-in-aws-lambda-with-postgresql-nokogiri
Ruby in AWS Lambda with PostgreSQL / Nokogiri > apparently so
December 28, 2018 - FROM lambci/lambda:build-ruby2.5 RUN yum install -y \ https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-6-x86_64/pgdg-redhat10-10-2.noarch.rpm RUN sed -i "s/rhel-\$releasever-\$basearch/rhel-6.9-x86_64/g" "/etc/yum.repos.d/pgdg-10-redhat.repo" RUN yum install -y postgresql10-devel RUN gem update bundler CMD "/bin/bash"
🌐
Docker
hub.docker.com › layers › lambci › lambda › build-ruby2.7 › images › sha256-7ca195b86ba116f5a4f43865ffbeeac534c72eb9a66bdba517c5fc53ef5c9c2f
Image Layer Details - lambci/lambda:build-ruby2.7
Welcome to the world's largest container registry built for developers and open source contributors to find, use, and share their container images. Build, push and pull.
🌐
GitHub
github.com › lambci › yumda › blob › master › amazon-linux-2 › build › specs › lambda2 › ruby.spec
yumda/amazon-linux-2/build/specs/lambda2/ruby.spec at master · lambci/yumda
serialize and de-serialize most Ruby objects to and from the YAML format. ... Provides telnet client functionality. ... # make symlinks for io-console and bigdecimal, which are considered to be part of stdlib by other Gems · mkdir -p ...
Author   lambci
🌐
GitHub
github.com › td-berlin › lambda_ruby_bundler
GitHub - td-berlin/lambda_ruby_bundler: Bundler for Ruby applications running on lambda
LambdaRubyBundler is a command-line tool for packaging Ruby applications for AWS Lambda. Most notably, it properly compiles dependencies with C extensions, using a custom Docker image based on lambci/lambda:build-ruby2.5.
Author   td-berlin
Top answer
1 of 2
10

Running ldd on gems/oj-2.18.5/lib/oj/oj.so clarifies the first error. the problem is not that oj.so does not exist, but that libruby.so.2.5 does not exist. The second problem is that the current Ruby lambda has glibc version 2.17, where AWS-linux comes with glibc version 2.25.

The fundamental problem here is that you need to install your gems on a system identical to the one they will run on if you have native dependencies. The best way I've found to do this is using docker. https://github.com/lambci/docker-lambda has docker images of lambda installs.

For ruby, do the following:

  1. Build the docker ruby image
  2. Copy your gemfile into a pristine directory
  3. From that directory, run: docker run -v "$PWD":/var/task --entrypoint bundle lambci/lambda-base:ruby2.5 install --path=/var/task

This will give you a folder called ruby with a version dependencies compatible with lambda.

If you are planning to use this output with lambda layers, keep in mind that the file structure produced by bunlde is ruby/2.5.0/... and it needs need to be ruby/gems/2.5.0/... before you upload it.

2 of 2
6

The problem is precisely as you stated, to summarize it: native extension gems need to be built in the same environment as they will be run. Therefore, before uploading the vendor to aws, we must install the gems in an environment which is similar to that of lambda.


To build your vendor/bundle in a way compliant with lambda, use the following docker container:

docker run --rm -v "$PWD":/var/task lambci/lambda:build-ruby2.7 bundle install --deployment

This will give you a working vendor/bundle dependency.

You can run another container to check if the function works:

docker run --rm -v $PWD:/var/task:ro,delegated lambci/lambda:ruby2.7 lambda_handler.lambda_handler

The full list of images is here: https://github.com/lambci/docker-lambda

The question's already 1 year old but I had to search for about 2 hours to solve the exact same problem with the gem nokogiri, so I thought this might be useful to somebody.

Hope this spares you from some googling hours!

Find elsewhere
🌐
Nisdom
nisdom.com › posts › 2019-05-14-aws-lambda-primer-with-ruby-redshift-s3
Aws Lambda Primer With Ruby using the RedShift, Secrets Manager and S3
May 14, 2019 - # my package build Makefile docker run -v $$PWD:/var/task -it --rm lambci/lambda:build-ruby2.5 \ /bin/bash -c 'yum -q -y install postgresql-devel && ...'
🌐
Medium
medium.com › @hsdeekshith › using-mysql2-gem-in-aws-lambda-for-ruby-7f9456bf2ca5
Using Mysql2 gem in AWS Lambda for Ruby | by deekshith h | Medium
August 26, 2019 - FROM lambci/lambda:build-ruby2.5RUN yum -y install mysql-devel RUN gem update bundlerCMD "/bin/bash"
🌐
DEV Community
dev.to › hint › ruby-on-aws-lambda-layer-dependencies-1ooe
Ruby on AWS Lambda: Layer Dependencies - DEV Community
August 11, 2020 - # Use AWS Lambda ruby2.7 build environment FROM lambci/lambda:build-ruby2.7
🌐
Pardner
blog.pardner.com › 2021 › 03 › the-quickest-easiest-way-to-develop-aws-lambda-functions-that-use-ruby-gems
The quickest & easiest way to develop AWS Lambda functions that use Ruby gems | Pardner’s Roundup
March 29, 2021 - The folder is just to hold the docker file you use when you need to build or rebuilt the Docker image you use for local AWS Lambda development. $ mkdir aws-lambda-docker $ cd aws-lambda-docker If using postgres (like the article above this was based on) it'll need extra stuff, this example below is just the simplest Dockerfile, to let you use ruby gems. # be sure the ruby version is same as what aws lambda currently offers # Dockerfile FROM lambci/lambda:build-ruby2.7 RUN gem update bundler CMD "/bin/bash"
🌐
Docker
hub.docker.com › layers › lambci › lambda › build-ruby2.5 › images › sha256-092a96a3e6a97f8660ced07e65e341d33114f11161fe7e1e87165e69213f4cc2
Image Layer Details - lambci/lambda:build-ruby2.5
Welcome to the world's largest container registry built for developers and open source contributors to find, use, and share their container images. Build, push and pull.
🌐
Northsail
northsail.io › articles › aws-lambda-ruby-2-7-pg-gem-libldap-error
AWS Lambda Ruby 2.7 Pg Gem LibLDAP Error | Northsail
June 22, 2021 - We are assuming for brevity that you have an established process in place to deploy your Ruby Lambda function. We leverage Serverless in our all AWS Lambda projects, which handles the heavy lifting of deployments for us. FROM lambci/lambda:build-ruby2.7 RUN yum install -y postgresql postgresql-devel RUN gem update bundler CMD "/bin/bash"
🌐
Medium
blog.francium.tech › deploying-ruby-to-aws-lambda-function-and-layers-a6c00b53a5d0
Deploying Ruby Script to AWS Lambda Function and Layers | by Navarasu | Francium Tech
February 7, 2019 - For Node.js, Python, and Ruby functions, you can develop your function code in the Lambda console as long as you keep your deployment package under 3 MB. Considering the gems with native extension, I prefer to install and build them using the ...
🌐
Amazon Web Services
docs.aws.amazon.com › aws lambda › developer guide › building lambda functions with ruby
Building Lambda functions with Ruby - AWS Lambda
Runtime: Choose Ruby 3.4. Choose Create function. The console creates a Lambda function with a single source file named lambda_function.rb. You can edit this file and add more files in the built-in code editor. In the DEPLOY section, choose Deploy to update your function's code.
Top answer
1 of 4
15

Do you have libpq.so.5 on your lib folder?

Your error is saying that did not find libpq.so.5 on the $PATH, in AWS Lambda the folder lib is automatically loaded on the path, thus, you just need to have this file there.

Executables created outside the lambda world do not run on Lambda, furthermore, you need to compile the executables by your own on a Lambda image. This is an example in how to do that:

Gemfile

source "https://rubygems.org"

gem "pg"
gem "mysql2"

handler.rb

require 'pg'
require 'mysql2'

def run(event:, context:)
  {
    postgres_client_version: PG.library_version,
    mysql_client_version: Mysql2::VERSION
  }
end

Dockerfile

FROM lambci/lambda:build-ruby2.5

RUN yum install -y postgresql postgresql-devel mysql mysql-devel
RUN gem update bundler

ADD Gemfile /var/task/Gemfile
ADD Gemfile.lock /var/task/Gemfile.lock

RUN bundle install --path /var/task/vendor/bundle --clean

This is going to build your image, then run it to generate the PG and MYSQL executables, then copy it to your lib folder.

build.sh

#!/bin/bash -x
set -e

rm -rf lib && rm -rf vendor && mkdir lib && mkdir vendor

docker build -t pg_mysql_layer -f Dockerfile .

CONTAINER=$(docker run -d pg_mysql_layer false)

docker cp \
    $CONTAINER:/var/task/vendor/ \
    ./

docker cp \
    $CONTAINER:/usr/lib64/libpq.so.5.5 \
    lib/libpq.so.5

docker cp \
    $CONTAINER:/usr/lib64/mysql/. \
    lib/

docker rm $CONTAINER

After running ./build.sh it is going to generate the folder lib and vendor with all you need, now you just need to deploy your lambda function.

To test locally you can run: docker run --rm -it -v $PWD:/var/task -w /var/task lambci/lambda:ruby2.5 handler.run

It is going to return something similar to this:

REF: https://www.stevenringo.com/ruby-in-aws-lambda-with-postgresql-nokogiri/

REF: https://www.reddit.com/r/ruby/comments/a3e7a1/postgresql_on_aws_lambda_ruby/

2 of 4
2

There are few good plugins to manage dependencies for AWS lambda. serverless-ruby-layer for ruby and serverless-python-requirements for python.

For your ruby case, you can use serverless-ruby-layer simply by adding plugin related config to your serverless.yml.

service: using-docker-yums

plugins:
  - serverless-ruby-layer

custom:
  rubyLayer:
    use_docker: true
    docker_yums:
      - postgresql-devel
    native_libs:
      - /usr/lib64/libpq.so.5

provider:
  name: aws
  runtime: ruby2.5

functions:
  hello:
    handler: handler.hello

And you need to install the plugin with the below command inside your serverless project folder,

sls plugin install -n serverless-ruby-layer

Now running sls deploy will automatically deploy the gems and libs to the layer.

Check out this example here in the docs

🌐
GitHub
github.com › lambci › lambci › issues › 11
Add Ruby support · Issue #11 · lambci/lambci
$ ./build-ruby.sh + rm -rf /tmp/maisi_website + git clone --depth 1 https://github.com/ksaynice/maisi_website.git /tmp/maisi_website Cloning into '/tmp/maisi_website'... fatal: could not read Username for 'https://github.com': No such file or directory Command "./build-ruby.sh" failed with code 128 ... @ksaynice you shouldn't be cloning the repo explicitly – as the comment in that file says, I'm only cloning that repo as an example. If you set your repo up with LambCI, then you'll already be in the repo directory when the build starts.