Monday, 7 October 2019

Packaging Ruby Code for AWS with Native Extensions

Back in November 2018, AWS announced support for running Ruby code in AWS Lambda.

AWS provides advice on how to package your Ruby code to deploy it as a Lambda function, and assuming a project laid out like this...
...the advice is as simple as:
  1. run bundle install
  2. zip up everything

As a build script, this could be:
set -euo pipefail
bundle install --deployment --path vendor/bundle --without development
zip -qr *.rb vendor

This works fine for trivial applications, but when your function contains gems with native extensions, we need to take a more sophisticated approach.

The lambda ruby runtime

When using gems with native extensions, the native extensions need to be compiled to be compatible with the configuration of the ruby runtime where these gems will be used.

The configuration of the ruby runtime you have installed on your development machine or CI/CD server probably doesn't match the runtime used by the AWS Lambda service, because it's likely to have been configured differently at build time.

lambci/lambda docker images

Fortunately, the lambci/lambda docker images are built from production AWS Lambda instances to mimic the production AWS Lambda runtimes in a docker container. These images are also used by Amazon's own SAM command line tool to run lambda functions locally.

These images are also available in build variants that are pre-configured with tools such as gcc that are needed to build native extensions.

A better build script

Using docker and the lambci images, we can create a better build script that leverages docker to ensure that native extensions are built in a way that is compatible with the AWS Lambda runtime.

First, we add a Dockerfile to the root of the project:
FROM lambci/lambda:build-ruby2.5
RUN chown 1000 /var/task

USER 1000

WORKDIR /var/task

COPY Gemfile Gemfile.lock /var/task/

RUN bundle install --deployment --path vendor/bundle --without development

COPY *.rb /var/task/

When built, this Dockerfile will configure an image containing your lambda code, and it's required gems, built to be compatible with the AWS Lambda runtime, using the lambci/lamba:build-ruby2.5 image as a base.

Now we adjust the build script to build the docker image and zip up the files from within the image:
set -euo pipefail
docker build --tag my-lambda-image .
docker run --volume "$PWD:/src" --workdir /var/task --user "$UID:$GROUPS" --rm my-lambda-image sh -c "zip -qr /src/ *.rb vendor"
This builds an image based on the Dockerfile above, then executes the zip command from within a container using that image.

The result is a zip file ( that contains the ruby code and all gems required for deployment to AWS Lambda.

Notes about the Dockerfile

  1. Most steps are performed as a non-root user, because bundle install does not like being run as root. The actual user id used is not important as all ownership information is lost when the zip file is created.
  2. Rather than copying all the source code into the container, the Gemfile and Gemfile.lock files only are copied in first, then bundle install is run. Doing this enables docker to cache the build steps all the way to the final COPY command if the Gemfile and Gemfile.lock files have not changed. This reduces the build cycle time greatly if the source code is changing, but the required gems are not.


The lambci/lambda docker images can be used to provide a build environment that closely mimics the actual AWS Lambda runtime.

These can be used in a build script to ensure that gems with native extensions are built in a way that is compatible with the AWS Lambda runtime.

The Dockerfile leverages docker's build caching system to allow the lengthy bundle install step to be skipped when the set of required gems have not changed.