How to create Python sandbox archive for AWS Lambda
AWS Lambda contain now 1067 Python libraries that we can use in our programs. The number is big and small at the same time. It should give us flexibility in writing apps but same time is limitation – there are many non-standard libraries that are better replacement for default ones. I will show you how to create Python application sandbox and then ZIP archive for AWS Lambda that will contain libraries not available by default so you can use them in your serverless application.
Using this application I’ve generated list of available libraries for Python 2.7 and you can check the list here.
Serverless applications idea is that we don’t have access to operating system. We just run our code in own sandbox. Therefor we can’t just install new package if we miss it. Solution is providing ZIP archive with code of our application and python environment that have all non-standard libraries inside. Let me show you how to do this.
Don’t create Python sandbox just on any computer
You can’t just use your PC or MAC to create python environment and import it to Lambda. Let me quickly explain you why.
Libraries are not just text files with code or binaries with compiled code you use issuing import statement. The functions you call in your app depends on python libraries which depends on system or other app libraries installed on your operating system. That means three major things needs to be compatible
- Hardware platform – Most computers use x86 architecture but you know that’s not the only one available on market, right? In example ARM processors are more and more popular thanks to cheap platforms like RaspberryPi, Apple use dedicated hardware for their phones, tablets etc. All those different processors require applications compiled specifically for them.
- Operating system – if you use Windows you can’t run code compiled for other operating systems, on MacOS you won’t be able to use .dll’s etc.
- Library version – functions implemented in different version of same library may not be compatible with each other. Nobody said backward compatibility is a must for developers.
If we prepare correct package on Mac and import it to Lambda it won’t work. We will see following errorm essage saying that binary library required for one of our module is not compatible with operating system beneath the Lambda
Unable to import module 'MyFirstLambdaApp': /var/task/cryptography/hazmat/bindings/_openssl.abi3.so: invalid ELF header
That means we have to create package on same platform as used by Lambda
Create python sandbox with libraries
Let me give you easy step-by-step instruction of how to create the sandbox environment:
Step 1 – Create EC2 instance with operating system same as one used by AWS Lambda. Official documentation lists AMI which was used to build Lambda and make the instance accessible from Internet via SSH so we can perform next steps. At the moment of writing this post AMI used for Lambda is amzn-ami-hvm-2016.03.3.x86_64-gp2 but always check if never version was not released. Use t2.micro instance so it’s cheap or even free for you if you are still Free Tier eligible. Don’t update packages in the instance.
Step 2 – Install development tools and some other system libraries. You will need it later to install python modules
[ec2-user@ip-172-31-44-79 ~]$ sudo yum groupinstall -y "development tools" [ec2-user@ip-172-31-44-79 ~]$ sudo yum install -y zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel expat-devel wget
Please note that for some python modules to compile you may need additional libraries not listed here. Pay attention to errors in Step 4 so you know what else you need to install.
Step 3 – Create python virtual environment using your future Lambda project name
[ec2-user@ip-172-31-44-79 ~]$ virtualenv CertSigningLambda
It will make python structure in folder named same as project you specified. Now activate the the environment using command in directory structure that you just created
[ec2-user@ip-172-31-44-79 ~]$ source CertSigningLambda/bin/activate (CertSigningLambda)[ec2-user@ip-172-31-44-79 ~]$
Notice the change of prompt. You can always leave sandbox:
(CertSigningLambda)[ec2-user@ip-172-31-44-79 ~]$ deactivate [ec2-user@ip-172-31-44-79 ~]$
Step 4 – install required additional python modules using pip tool. You can create text file with lists of libraries (and their versions if you required not latest one). My requirements.txt file is:
pyOpenSSL==17.0 parse
Execute the installation and look for any errors. At this stage you will see if you have all system libraries installed.
(CertSigningLambda)[ec2-user@ip-172-31-44-79 ~]$ pip install -r requirements.txt
When pip install all specified modules with no errors you will be able to run your python application in the sandbox!
Create package for AWS Lambda
Now it’s time to create zip package with sandbox so we can use it in AWS Lambda. Because at multiple project stages I test code directly on AWS Lambda, not locally on my computer, I prefer to have zip file with python sandbox and add source code of my application to archive every time I need to test it. This way I don’t need to keep EC2 instance all the time – I will only need it if new library is required for my application.
Creating the zip archive of your python sandbox is really easy
Step 1 – Copy python and dependent system libraries from sandbox to new folder. We are not preserving directory structure but copy content of specified folders to root of new directory.
(CertSigningLambda)[ec2-user@ip-172-31-44-79 ~]$ mkdir LambdaSandbox (CertSigningLambda)[ec2-user@ip-172-31-44-79 ~]$ cp -rf CertSigningLambda/lib/python2.7/site-packages/* LambdaSandbox/ (CertSigningLambda)[ec2-user@ip-172-31-44-79 ~]$ cp -rf CertSigningLambda/lib64/python2.7/site-packages/* LambdaSandbox/
Remember you have to copy files from both lib and lib64 directories!
Step 2 – Create ZIP archive. One important thing – do it from inside of the folder where you just copied libraries from your sandbox! Path to libraries must start in root of your archive.
(CertSigningLambda)[ec2-user@ip-172-31-44-79 ~]$ cd LambdaSandbox/ (CertSigningLambda)[ec2-user@ip-172-31-44-79 ~]$ zip -r ../CertSigningLambda.zip .
Now you have copy of sandbox ready for AWS Lambda
Add source code to your archive
Now you have to add source code of your python application into archive. You can either copy it to EC2 instance or copy ZIP archive to your local computer. Again it’t important that you place structure of your program files in root directory of the archive. It will be mixed with modules and libraries copied from sandbox but Lambda engine will know how to use it.
Assuming that you copied zip archive to root folder of your python project and you application contain only one file you will add it to archive using command
localhost:CertSigningLambda developer$ zip -r CertSigningLambda.zip CertSigningLambda.py
Now upload archive to AWS Lambda and run your application. If you don’t see any error saying either cannot import library or missing library then your archive contain all required python libraries and dependent files.
Summary
If you want to use library not included in default AWS Lambda environment you must create your own sandbox on dedicated EC2 instance created from same AMI that Lambda is using. Then you need to install in sandbox all additional libraries that are not available in Lambda and their dependencies using pip tool. Now you need to prepare the ZIP archive from modules and libraries from your sandbox – remember that everything from site-packages directories need to be places in root of the archive. Then you can add source code of your application and any directory structure you use to the archive, also to its root.
The task is not very complicated and we can easily write bash script for automation. But it can take some time until all required perl modules are provided.
If you are porting existing app to AWS Lambda I recommend first to create empty python files with all import statement from all your files and empty function that Lambda will call. This way you can step-by-step, error-by-error create archive with virtual sandbox that include all non-standard libraries you need.