Menu Close

AWS Lambda – Installing libraries and binaries

AWS Lambda - Instalando librerías y binarios

1. Use Case

AWS Lambda is a very versatile service that provides the ability to develop applications at a low computational cost because they do not require an instance for their executions and only cost based on their calls.

There will be times when we want to use resources that are not directly in Lambda and are crucial for the operation of our solution, for this we will use the Lambda layers where we can install these packages and then use them in our solution as if they had always been installed.

Exemplification: Python libraries and binary files

Consideration: Although it is exemplified by Python, we can carry out this process for any programming language accepted by Lambda.

2. Pre-requirements

2.1. Docker

AWS Lambda runs on a Linux image that contains the dependencies to run your code. But if you are using Windows or OSx we can simulate an AWS Lambda environment using a Docker container.

You can get it through the following link..

Note: Try to download Docker desktop.

 

3. Image settings

3.1. Creating a Lambda image

The first thing we will do is go to our terminal and refer to a folder where we want to store this image. Once there, we will proceed to run the following command:

docker pull lambci/lambda:build-python3.6

Note: In this case we are downloading the requirements to proceed to create a Lambda image with Python 3.6, but it is possible to use other languages ​​or other versions.

 

3.2. Simulate the environment

After doing this, we proceed to run the image using the following command, which uses the current folder to simulate a bash console of our image.

docker run -v “%cd%”:/var/task -it lambci/lambda:build-python3.6 bash

OSx: In case you are developing on OSx change the value %cd% by $PWD.

3.3. Package installation

Having our bash console working, we can proceed to the installation of the packages that we consider appropriate. For this, we will create a folder where we will install the libraries and then package the folder and upload it as a layer.

AWS recommends using the name of the language used for the folder, in our case python. For this example we will install a binary FFMPEG file, which you can find in this link. The important thing is that it is the executable of the program.

 

3.3.1. Step 1: creating folder

  • We will use the following command to: mkdir python

 

3.3.2. Step 2: Adding binaries

Any executable binary that we want to add to our image must be added to the main directory. For this example, we will download the latest version of FFMPEG, unzip the file, and copy the ffmpeg and ffprobe files to the folder we created in the previous step.

 

3.3.3. Step 3: Install other language resources

Through our bash console we will enter the folder and install the dependencies in it. For example if we wanted to install Pandas we would do it like in a normal Python installation but adding the -t flag to specify where the installation will be done along with ./ to refer to the current folder.

  • cd python
  • pip install pandas -t ./

3.3.4. Step 4: Compress resources

In order to be used as a layer, we must compress the files into a .zip file. AWS requires that these files have all the permissions, not only read, unfortunately if you are in Windows it does not matter that you change the configuration of each file because compressing it will lose this information.

To solve this you can use the zip_chmod777.py file, with which you can modify each internal file and give it the permissions.


Archive zip_chmod777.ph
from zipfile import ZipInfo, ZipFile, ZIP_DEFLATED

def access_handler(name):
    old_zipname = name
    new_zipname = ‘layer_’ + name  

    old_zip = ZipFile(old_zipname,’r’)
    new_zip = ZipFile(new_zipname, ‘w’, compression=ZIP_DEFLATED)

    for file_name in old_zip.namelist():
        zip_info = ZipInfo(file_name)
        zip_info.compress_type = ZIP_DEFLATED
        zip_info.create_system = 3 # Specifies Unix
        zip_info.external_attr = 0o777 << 16 # Sets chmod 777 on the file
        new_zip.writestr(zip_info, old_zip.read(file_name)) # You have to write the file contents in with ZipInfo

    old_zip.close()
    new_zip.close()

if __name__ == “__main__”:
    name = input(‘Zip File name: ‘)
    if ‘.zip’ not in name:
        name += ‘.zip’
    access_handler(name)



In general terms, what is done in this function is to receive the name of a compressed zip file and create a new one with the prefix
layer_. Then to each file within the original zip, 3 points are specified, the compression type, the operating system (which we will use 3 to reference Unix) and we will add an external attribute where the code present there 0o777 << 16 gives it all the permissions to files. Finally the new file is saved.

We can run this file by calling it through a console or by double-clicking it from the directory that stores our zip.

Important: Once this is done, delete the original zip and rename the new zip to be the AWS recommended way.

 

4. Layer settings

4.1. Storing the resource in the cloud

There are certain policies on how much is the appropriate weight to upload the file directly to the layer that we will create in Lambda or if we must store it in a Bucket in S3. For this we already had a Bucket configured where we were performing audio to text conversion. Right there we created a folder called libs (conveniently named, there is no requirement that prevents naming it as we want) and we uploaded the compressed file there.

Important: Copy the link to the file as we will need it later.

4.2. Creating a Lambda Layer

  • The first thing will be to go to our AWS console and enter Lambda.

  • Once inside, we will enter the Layers or Layers section.

 

  • Then we will create a new layer where a form will be displayed


  • Here, we will enter a representative name of the layer and, although it is optional, it is recommended to enter some description because we can make versions of the layer and the description will allow us to recognize which version of the layer we need to use. Then we select the option Upload a file from Amazon S3 and we put the link of the file that we uploaded in the previous section. Last but not least, we select the runtime / s compatible with our package, in this case we will put Python3.6.

4.3. Add the layer to the Lambda function

  • Finally, in order to use these libraries we will go to the function that we are implementing


  • We select the part that says Layers and in the lower menu
  • Reminder: The resource will be available and visible only for functions that share the runtime we defined above.
    That’s it, we can already use the resource in our function, particularly for the example we have done, we can use the ffmpeg binary from the local folder in lambda

 

  • FFMPEG = ‘/opt/python/ffmpeg’ And call it through a sub-process # INPUT: audio_filename.extension def convert(filename, extension=’mp3′):

 

  • CMD = “{0} -i {1} {2}”.format(FFMPEG,TEMP+filename,TEMP+filename.split(‘.’)[0]+‘.{0}’.format(extension)) process = subprocess.check_output(CMD, shell=True, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL)
    return process

5. References

Write us!

Posted in »Blog English