Home

Awesome

Simple AWS Lambda function to extract specific parts of the UK Met Office's UKV and MOGREPS-UK numerical weather predictions, compress, and save to S3 as Zarr.

The Lambda function itself is the lambda_handler function in metoffice_aws_lambda/app.py

Please note that this code is no longer being maintained or used (because it was too expensive to run thousands of long-running jobs on Lambda!) Instead we're using EC2. See the new metoffice_ec2 code, which supercedes metoffice_aws_lambda.

Installation on Linux

Install conda environment. Then find where Python is installed for your conda env (e.g. ~/ miniconda3/envs/metoffice_aws_lambda/bin/pip) and run:

~/miniconda3/envs/metoffice_aws_lambda/bin/pip install aws-sam-cli

Add this line to ~/.bash_aliases:

alias sam='python3 -m samcli'

And then load with source ~/.bash_aliases

Build and deploy with:

sam build
sam deploy --guided

SAM will then ask some questions. These are reasonable answers:

Stack name                      : metoffice-aws-lambda
Region                          : eu-west-2
Confirm changeset               : N
Allow SAM CLI IAM role creation : Y
Save arguments to samconfig.toml: Y

Configure AWS permissions

Go to the AWS Identity and Access Management (IAM) console, and attach policy AWSLambdaSQSQueueExecutionRole to the role for the metoffice-aws-lambda function.

Create bucket for storing NWPs

Create a bucket called metoffice-nwp (or change the value of target_bucket in app.py).

Configure AWS Simple Queue Service (SQS)

When the Met Office uploads new NWPs to S3, they also send a message to an AWS Simple Notification Service topic. It is possible to trigger lambda functions directly from SNS notifications. However, this results in the lambda function sometimes triggering too soon. This often means the lambda function will take a long time (300 seconds) to download the NetCDF file from S3; and sometimes means the lambda function cannot find the NetCDF file at all.

A solution is to set up a Simple Queue Service, and the the SQS to delay messages a little, to ensure that the NetCDF files are ready and waiting on S3 by the time our lambda function triggers.

Set up SQS as per the Met Office's instructions.

Additionally, set these config options on the queue:

Configure SQS to trigger lambda function

Go to the AWS Lambda console. Click 'Add trigger'. Add the SQS queue you created above. Set batch size to 1.

Deleting deployment