Awesome
AWS Lambda Debugger
Do you want to step through code running live in Lambda? Do you want to fix bugs faster? Do you want free pizza?
This project will help you with the first 2 questions. When you show it to a friend, you might get that 3rd one too :)
This is only for the AWS Node runtime. Node 4.3 is not supported.
Isn't this impossible?
No. Well, not anymore.
How?
Normally, debugging is a one hop process: developer's debugger connect directly to the process. This is impossible with Lambda.
However, we fork your code to a separate child process that is running in debug mode and connected to the original via an interprocess communication channel. The parent process opens 2 WebSockets as well: one to the child process' V8 Inspector and the other to a broker server, becoming a proxy between the 2 connections. Next, the developer connects a debugger to the broker server, which connects them to the proxy, which is connected to the child's debugger port. Now, you have a 3 hop chain like this:
Debugger <=> Broker <=> Proxy <=> Child
Once the developer's debugger connects to the broker server, the debug version of the handler is executed. The proxy and the child coordinate to shim the event, context, and callback. Result: the developer is connected to a live running Lambda function with full control.
Oh, and you only have to add one line of code at the end of your handler file(s)...
I want one!
Good. There are 5 steps:
- Deploy the broker server
- Add the proxy to your code
- Configure the proxy via environment variables
- Increase your Lambda timeout
- Use it!
Deploy the broker server
You should only need one of these for you or your team to start. The broker has been designed for multiple simultaneous sessions. We found a t2.small to be more than enough for starting out.
- Kick off an EC2 Amazon Linux Instance
- Attach Security Group
- exposing ports
8080
and9229
to0.0.0.0/0
- expose port 22 to YOUR IP
- exposing ports
- SSH in to the box
# Install Docker
sudo yum update -y
sudo yum install -y docker
sudo service docker start
# Run the Broker
docker run --name debug-broker \
-d -p 8080:8080 -p 9229:9229 \
--restart always \
trek10/aws-lambda-debugger
# To view logs
docker logs -f debug-broker
Advanced networking configuration info
Here's extra details about the port configurations:
- Lambda connects to
8080
. If your Lambdas are in the same VPC, you can configure the security group to just allow them in this side. - The default port for the V8 Inspector is
9229
. You can restrict access to this port based on where your developers are coming from.
Add the proxy to your code
Add the package to your repo:
npm install aws-lambda-debugger --save
Require the package at the very end of each file that contains a Lambda handler that you want to debug. Example:
module.exports.myHandler = (event, context, callback) => {
// put some code that you want to debug here
}
require('aws-lambda-debugger');
That's it!!!
How the magic works: By putting the require
call at the end of the module, it is the last thing to be evaluated before the module is loaded by Lambda. For whatever reason, Node provides access to the parent module and does not or can not stop us from mutating that parent module. This allows us to reach up into the parent and swap the handler before it is executed. The proxy code takes over and launches your handler in the child process instead.
Configure the proxy via environment variables
There are 3 magic environment variables that need to be set:
DEBUGGER_ACTIVE
: As long as this value is present and it is not 'false' or empty string, the proxy will do its job.DEBUGGER_BROKER_ADDRESS
: This is the IP address or domain for the broker server.DEBUGGER_FUNCTION_ID
: This is a unique "ID" of your own choosing (per function!) that is used by the broker to pair the debugger connection (this function ID is also part of the URL that the debugger connects to - see below) to the appropriate Lambda function. It does not need to be a UUID/GUID/etc. It just needs to be unique relative to any other functions that you or your teammates are using on the same broker instance.
Increase your Lambda timeout
Alter the timeout to 300 seconds to allow maximum debug time.
Use it!
-
Launch your Lambda function (from console, via CLI, etc)
-
Replace the
DEBUGGER_BROKER_ADDRESS
andDEBUGGER_FUNCTION_ID
in the following URL and paste it into Chrome.chrome-devtools://devtools/remote/serve_file/@60cd6e859b9f557d2312f5bf532f6aec5f284980/inspector.html?experiments=true&v8only=true&ws=[DEBUGGER_BROKER_ADDRESS]:9229/[DEBUGGER_FUNCTION_ID]
-
DEBUG!!!
Bonus! Use this bookmarklet to pull the URL from the Code tab for the function in the AWS console:
javascript:(function()%7Bconst obj %3D %7B%7D%3B document.querySelectorAll('.env-var-list .key-value').forEach(elem %3D> %7B if (elem.querySelectorAll('input%5Bplaceholder%3D"Key"%5D').item(0).value) obj%5Belem.querySelectorAll('input%5Bplaceholder%3D"Key"%5D').item(0).value%5D %3D elem.querySelectorAll('input%5Bplaceholder%3D"Value"%5D').item(0).value %7D)%3B const win %3D window.open(''%2C '_blank')%3B win.document.write(%60Debugger URL%3A chrome-devtools%3A%2F%2Fdevtools%2Fremote%2Fserve_file%2F%4060cd6e859b9f557d2312f5bf532f6aec5f284980%2Finspector.html%3Fexperiments%3Dtrue%26v8only%3Dtrue%26ws%3D%24%7Bobj.DEBUGGER_BROKER_ADDRESS%7D%3A9229%2F%24%7Bobj.DEBUGGER_FUNCTION_ID%7D%60)%7D)()
What's the catch?
There are a few catches/known issues:
- Multiple
console.log
calls close together sometimes causes them to be aggregated in a single CloudWatch Log entry - This only works with the AWS Node 6.10 runtime. We expect it to work with Node 8 whenever AWS offers support for it too.
- Chrome DevTools is the only debugger that we have proven to work. YMMV with your IDE.
- You pay for your debug time. No way around this. It is a running Lambda function after all.
context.callbackWaitsForEmptyEventLoop
is defaulted tofalse
. You can change it back totrue
(we even shimmed that!), but your function will run until it times out if you do.context.getRemainingTimeInMillis
is technically an approximation. We grab the remaining time and the current timestamp when the debugger connects and ship both to the child. The time delta is then subtracted from the original time. Since all times are retrieved inside of the Lambda, this should be a very close approximation.
Anything else I should know?
Functionally, this thing is complete. However it is still very new, so don't be surprised if something goes wrong. Feel free to file an issue. We're happy to take PRs too.
Items that need work still
- Tests: There are no tests. Because of all of the dark arts, eventing stuff, and the sockets, we didn't want to delay release because the tests are going to be hard to write. If you're a ninja at this, feel free to reach out.
- Improving logging in the broker server
- Bulletproof socket cleanup: We're still trying to make sure that all sockets are cleaned up as fast as possible. If you find any leaks, please let us know.
- Study memory usage in the broker.
Future ideas:
- Make things more configurable
- Web UI for broker server
- Get direct link as soon as Lambda connects to broker
- Track remaining time
Made with :gift_heart: and a pinch of black magic: by Jessica Ribeiro + Trek10
P.S.: We do AWS consulting and monitoring. Come talk to us.