Awesome
Running Haskell AWS Lambda Functions
This repository contains code and directions to run Haskell executables over AWS Lambda "serverless" infrastructure. This experiment was triggered by reading description of apex provide a wrapper to run Go code.
Howto - Automated Way
The main.hs
aims at automating the deployment of Haskell code packages to AWS Lambda:
- Manaing AWS Lambda functions and packages,
- Manage AWS Api Gateway endpoints to expose Lambda functions.
Interaction with AWS is done through the excellent amazonka package.
To build CLI program:
$ stack build
Creating an AWS Lambda package (requires docker):
$ ./main lambda build --build-target foo --source-directory foo/
This generates a lambda.zip
file in the current directory that contains something like:
$ unzip -l lambda.zip
Archive: lambda.zip
Length Date Time Name
-------- ---- ---- ----
1131 06-07-16 10:38 run.js
32648176 06-07-16 10:38 words
364776 06-07-16 10:38 libblas.so.3
5638304 06-07-16 10:38 liblapack.so.3
1186104 06-07-16 10:38 libgfortran.so.3
244904 06-07-16 10:38 libquadmath.so.0
-------- -------
40083395 6 files
Deploying an existing lambda.zip
file to AWS:
$ ./main lambda deploy --function-name foo
This creates function foo
that can be invoked manually (see below).
Deleting an API Gateway endpoint:
$ ./main api delete --endpoint fooAPI
Note that AWS has a rate limit on API deletions.
TODO
- Creating an API Gateway endpoint and linking it to Lambda function: The configuration and part of the code are ready but the process is a little bit involved hence requires more interaction with AWS
Howto - The Manual Way
Prepare environment
- Ensure access to AWS Lambda service:
- use
aws configure
to define credentials and region to deploy to - create or use role for executing code on AWS Lambda, e.g. something like
arn:aws:iam::259394719635:role/lambda
- use
Simple Build
-
Build docker container for building Haskell code that is supposed to be runnable on Amazon's Linux AMI
cd ghc-centos docker build -t haskell-centos . cd ..
-
Build haskell code:
docker run -ti -v $(pwd):/build -w /build --name=haskell-build haskell-centos stack build --allow-different-user ... CONTAINER_ID=$(docker ps -a | grep haskell-centos | head -1 | cut -d ' ' -f 1) docker run --volumes-from=$CONTAINER_ID busybox dd if=/build/.stack-work/install/x86_64-linux/ghc-7.10.3/7.10.3/bin/main > main
-
Build zip file containing Javascript wrapper and main app
zip lambda.zip run.js main
Complex Build
A lot of interesting pieces of code rely on external C libraries. For example, hmatrix relies on LAPACK and BLAS libraries for efficient matrix operations and compiled executable will need to be dynamically (or statically) linked to those libraries for proper execution. The above directions should be updated to take into account those extraneous libraries:
-
Add the needed libraries into the build container description, e.g.
RUN yum install -y lapack-devel blas-devel
-
Build code as before, using the custom image,
-
Export from container the needed libraries (the will be packed as part of the code shipped to AWS Lambda):
$ docker run --volumes-from=$CONTAINER_ID haskell-centos ldd /build/.stack-work/install/x86_64-linux/ghc-7.10.3/7.10.3/bin/main > libs ... extract list of library files to copy... $ for i in $(cat libraryFiles); do docker run --volumes-from=$CONTAINER_ID haskell-centos dd if=$i > $(basename $i) done
-
Modify the
run.js
wrapper to set correctlyLD_LIBRARY_PATH
environment:process.env['LD_LIBRARY_PATH'] = process.env['LAMBDA_TASK_ROOT'] ctx = context
-
Pack everything into
lambda.zip
: Javascript wrapper, libraries, executable...
Deploy to AWS Lambda
-
Create function on Lambda:
$ aws lambda create-function --function-name hello-test --runtime nodejs4.3 --zip-file fileb://./aws-lambda.zip --handler run.handle --role arn:aws:iam::259394719635:role/lambda { "CodeSha256": "QYKOebaDN/fqEzb1nmaV3ByNDZK3JvD0kWX6YQnPpjE=", "FunctionName": "hello-test", "CodeSize": 265356, "MemorySize": 128, "FunctionArn": "arn:aws:lambda:eu-west-1:259394719635:function:hello-test", "Version": "$LATEST", "Role": "arn:aws:iam::259394719635:role/lambda", "Timeout": 20, "LastModified": "2016-05-23T10:32:38.126+0000", "Handler": "run.handle", "Runtime": "nodejs4.3", "Description": "" }
-
Run function on Lambda:
$ aws lamdba invoke-function --function-name hello-test { "StatusCode": 200 } $ cat test "child process exited with code 0"
The provided main.hs
simply output its input to its output. There should be an execution trace in the logs hosted on CloudWatch:
Manifest
ghc-centos
: Docker container for building Haskell binaries compatible with Amazon's Linux AMI. Does not seem to be a good idea in general as there are quite a few differences between CentOS and Linux AMI, but in practice it kind works...run-tmpl.js
: Template Javascript wrapper to run binary in a child process. The$$main$$
string should be replaced by the name of the packed executable,test.js
: Javascript test wrapper, invoke the handler simulating what AWS Lambda doesstack.yaml
,main.cabal
,main.hs
: Basic structure for building Haskell code