Better Lambdas with pip packages in CloudFormation

5 min readFeb 15, 2021


After writing about how to create AWS Lambda Layers based on pip packages, I set out to make writing lambda functions in CloudFormation a bit easier. In the official CloudFormation AWS::Lambda::Function you either must upload your functions as zipfile to S3 and reference them from there, or, if you want them inline, you run into 3 large issues

  • You’re limited to 4096 bytes of code (meaning in my experience that as the code gets more complex, you start saving on comments, function names, error checking, etc, making the whole thing even harder to read and maintain).
  • You’re limited to a single file. Maybe not so bad if you only have 4096 bytes, but still not good for readability
  • You are limited to the builtin packages cfnresponse and boto3; installing packages from pip is not that easy. This limitation is mitigated by the pip layers mentioned before.

Now, let me lead of by saying that for more complex environments (production environments), I definitely recommend having uploading packages through the S3 way. At this time you should also have multiple stacks (or even multiple accounts), and your own tooling for doing stack stuff. However for someone who just experiments a bit on their own account with a small project now and then, it’s nice to have self-contained stacks, that do the job from start to end.

So in this article I introduce a way to use Custom Resources to create a function from inline python code, of arbitrary size (note: there is a limit of 51200 bytes for the template as a whole, after which you will still need an S3 spot to upload your template to CF, but this at least gives a bit of leeway).

To start off, I use a small python tool to build my stack. It allows me to do something like:

__INCLUDE__0: baseresources.yaml
__INCLUDE__1: lambdas.yaml

Type: AWS::Lambda::Function
FunctionName: testlambda
ZipFile: |

Note that this tool is far from perfect, however it allows me to reuse code and write the python in .py files, which means that the editor helps me.:

Now, using the example before, as soon as gets larger than 4096 bytes (which is about 2 pages of code, not that much), the stack update will fail.

It seems that the 4096 bytes limit is arbitrary; it’s totally possible to send larger documents to Custom Resources, so we could actually make a custom resource that receives this code, and then creates a lambda function out of it. For the past days I’ve been trying to do just that, however both having the custom resource create the lambda directly, and having the custom resource update the code of some existing lambda failed, because of my uneasiness with lambdas that can potentially escalate privileges. After some more experimentation, the (more secure) solution seems to be to put all the code in a Lambda Layer. Since lambda layers are versioned (as far as I know, it’s not possible to tell a function to always run the latest layer version) an attacker would need to gain access to more than just the Custom Resource lambda in order to mount an attack.

So the result is a Custom Resource that creates a lambda layer. The lambda layer contains both the packages and the code. The lambda function itself contains no code at all — we can use the Handler property of a function to directly call code in the layer. So to get a (very uninteresting) lambda — note that we give it so few rights that it cannot even log execution:

Type: AWS::IAM::Role
Version: 2012-10-17
- Action:
- sts:AssumeRole
Effect: Allow
Type: AWS::Lambda::Function
ZipFile: '# NOTE: this file is not used, all code is in layer'
Description: Test Function
FunctionName: testlambda
Handler: testlambda.handler
MemorySize: 128
Role: !GetAtt TestLambdaRole.Arn
Runtime: python3.8
- !Ref TestLambdaLayer
Type: Custom::CodeLayer
ServiceToken: !GetAtt CodeLayerLambda.Arn
LayerName: TestLambdaLayer
- numpy==1.20
Files: |
import numpy as np
def handler(event, context):
print(np.arange(10) ** 2)

It creates a lambda function with an with the string # NOTE: this file is not used, all code is in layer. Obviously you can put here whatever you want, I usually opt for just ... although this text is better in case I forget later what I did. The TestLambdaLayer is where the magic happens; the custom resource builds a layer with the file, with the numpy package installed, and the TestLambda function runs directly the handler() function from the file.

Note that any update to the code will result in a new layer version being created, then the TestLambda function being updated to use the new layer version. In case of a single application consisting of 4 of 5 lambda functions, one could create a single layer with all the code, and each lambda function just pointing to a different entry point (Handler).

The CloudFormation code to create the custom function is below (note that it uses the __INCLUDE__: syntax from the file above; this is just for readability, and things will work just as well if you manually copy the code into the codelayer.yaml file (and indent it far enough). The code has the following features:

  • Fits in 4096 bytes (so can be used inline in a CF template)
  • Uses a 300 second timeout and 1024MB of memory limit. Feel free to expand/shrink these limits to your use. At today’s pricing, a 300 second run for a 1024MB lambda function costs about $0.005, or half a cent.
  • Installs packages from pip (you should be able to use any syntax that is valid for python -m pip .... Note, packages have to be given as array, with multiple packages in multiple array items. It will auto-install the packages for the python version that the lambda runs in (python 3.8 for the code below). In almost all cases this will install a version for all recent 3.x python versions; if it’s important, you can change the runtime on the CodeLayerLambda.
  • Installs files as given through key-value in the Files section. Creates directories where necessary, so you can do something like {"Files": {"test/": "import unittest\n..."}} .
  • Allows creation of python and non-python files. Python files are syntax-checked, and deploy fails if python -m py_compile #filename# fails for a file with extension .py. Obviously it’s easy to remove this check below. Non-python files are limited to text-files for now.
  • Note: Security was not a big design requirement here; it’s assumed that only admins can run this function. I don’t see any obvious ways how a bad actor can do something bad within the lambda environment while running this, but sanity checking all the code in a way that I would like to do, makes the lambda (much) bigger than 4096 bytes.




💚 Computer Science, 3d printing, quantum mechanics, nature, (water) sports. Exploring myself, and writing about it all.