Python programming blog

I help Python Web Developers to design, build, debug, deploy, and test API efficiently, so they can be more productive and independent

Deploy Django App on AWS Lambda using Serverless (Part 1)

dj_sls_lambda.png

# BLUF

As a follow-up to a [post](https://dev.to/vaddimart/top-6-questions-people-ask-about-django-apps-in-a-cloud-31bj) where we looked at the most common questions about Django in the Cloud, now, I'd like to help you deploy your Django App on Amazon Web Services and make you more independent from other developers like DevOps and CloudOps Engineers. There are many options for doing that but I'd like to show one of them and I hope that in the end you will be able to deploy your Django App on AWS Lambda using Serverless.

I was motivated by [Daniil Bratchenko's](https://www.linkedin.com/in/bratchenko/) article [Don’t Let Software Vendors Dictate Your Business Processes](https://daniil-bratchenko.medium.com/dont-let-software-vendors-dictate-your-business-processes-6bee1dd78234) to start writing this blog post.

It is so hard to find Software that will fit all your business processes as all companies are unique. This is why many companies have decided to set up dedicated teams building Software for their specific business processes and needs. From my personal point of view, Django App on AWS Lambda using Serverless is a good solution for cases like that.

Also, you can use this approach for prototyping your projects running them at their early stage.

There are a few advantages and disadvantages of using this approach.

**Advantages of using AWS Lambdas**:

* cost (AWS Lambda is cheaper comparing to AWS EC2);

* simplicity in running and maintaining;

* scalability;

* quick deployment.

**the disadvantages**:

* AWS Lambda requires some extra time to run your App;

* size limit for deployment package;

* API Gateway limitation (30-sec timeout, 6 Mb response body size);

* it might cost more than AWS EC2 if there are too many requests.

## Prepare AWS infrastructure

Probably, you are aware of a variety of AWS services required for web applications. In order to deploy a Django project on AWS Lambdas you should prepare your AWS infrastructure.

There is a list of AWS services I use for my Django project:

1. Lambdas to run our wsgi application

2. API Gateway to handle HTTP request and send them to Lambdas

3. S3 buckets for Lambda deployments and storing static files

4. CloudFront distribution for serving static files from S3 bucket

5. RDS for a database (I use Postgres)

6. VPC with subnets

7. EC2 for Security Groups

8. IAM for roles and policies

9. CloudWatch for logs

AWS Lambdas, API Gateway will be created automatically by Serverless. I will try to walk you though the process of creating all the necessary AWS resources in my following blog posts.

## Create a Django project

Django `startproject` command allows us to create a simple Django project, in addition to that, there are some great Cookiecutter projects that can help you start your project easily (For example [Cookiecutter Django](https://github.com/pydanny/cookiecutter-django)). I use default `django-admin startproject` cli command in this example.

```bash

pip install django

django-admin startproject django_aws_lambda

```

## Configure requirements

There are many options to store your project requirements, for example `requirements.txt`, `Pipfile`, `pyproject.toml`. You can one of these options. I'm using `requirements.txt` here.

* create `requirements.txt` file in a root directory of the project

* add the following libraries to `requirements.txt` file:

```

boto3==1.17.17

Collectfast==2.2.0

Django==3.1.7

django-environ==0.4.5

psycopg2-binary==2.8.6

Werkzeug==1.0.1

```

* create and activate virtual environments

> Choose your preferred tool for managing virtual environments (like conda, pyenv, virtualenv, etc.)

* install requirements

```bash

pip install -r requirements.txt

```

## Create `hello` Django app

* create app using `startapp` Django command

```bash

python manage.py startapp hello

```

* create `templates` folder

```bash

mkdir templates

```

* create `index.html` file in `templates` folder with the following lines:

```html

{% load static %}

<!DOCTYPE html>

<html lang="en">

<head>

<meta charset="UTF-8">

<title>Greeting</title>

</head>

<body>

<div>

<h1>Hello {{ name }}</h1>

<img src="{% static 'django.jpeg' %}" alt="Django" style="width: 20%">

</div>

</body>

</html>

```

* create folder `static` in the root directory of the project

```bash

mkdir static

```

* add an image file to `static` folder, for example `django.jpeg`

* update `hello/views.py`

```python

from django.shortcuts import render

# Create your views here.

def hello(request, resource=None):

return render(request, "index.html", {"name": resource or 'World'})

```

## Configure environments variables:

* create `.env` file in the root directory of the project

* configure the following variables:

```dotenv

STAGE='production'

DB_HOST=<your database host>

DB_USER=<your database user name>

DB_PASSWORD=<your database password>

DB_NAME=<your database name>

DJANGO_SECRET_KEY=<some django secret key>

AWS_S3_CDN_DOMAIN=<your Cloud Front distribution, like: `<distribution id>.cloudfront.net`>

AWS_S3_REGION_NAME=<your AWS region>

AWS_STORAGE_BUCKET_NAME=<AWS s3 bucket for static files with punlic policies>

DEPLOYMENT_BUCKET=<AWS s3 bucket for deployment>

AWS_KEY_ID=<your AWS Key Id>

AWS_SECRET=<your AWS Secret>

DJANGO_ADMIN_URL=<Django admin url>

DJANGO_ALLOWED_HOSTS=<list of allowed hosts separated by coma>

```

## Create configuration for local development and production

* update `settings.py` in `django_aws_lambda` folder with the following lines:

```python

"""

Django settings for django_aws_lambda project.

Generated by 'django-admin startproject' using Django 1.11.29.

For more information on this file, see

https://docs.djangoproject.com/en/1.11/topics/settings/

For the full list of settings and their values, see

https://docs.djangoproject.com/en/1.11/ref/settings/

"""

from pathlib import Path

import environ

ROOT_DIR = Path(__file__).resolve(strict=True).parent.parent

env = environ.Env()

READ_DOT_ENV_FILE = env.bool('DJANGO_READ_DOT_ENV_FILE', default=True)

if READ_DOT_ENV_FILE:

env.read_env(str(ROOT_DIR / '.env'))

# Quick-start development settings - unsuitable for production

# See https://docs.djangoproject.com/en/1.11/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!

SECRET_KEY = env('DJANGO_SECRET_KEY', default='<some-secured-key>')

# SECURITY WARNING: don't run with debug turned on in production!

DEBUG = False

ALLOWED_HOSTS = env.list('DJANGO_ALLOWED_HOSTS', default=['127.0.0.1', 'localhost'])

INSTALLED_APPS = [

'django.contrib.admin',

'django.contrib.auth',

'django.contrib.contenttypes',

'django.contrib.sessions',

'django.contrib.messages',

'django.contrib.staticfiles',

'hello',

]

MIDDLEWARE = [

'django.middleware.security.SecurityMiddleware',

'django.contrib.sessions.middleware.SessionMiddleware',

'django.middleware.common.CommonMiddleware',

'django.middleware.csrf.CsrfViewMiddleware',

'django.contrib.auth.middleware.AuthenticationMiddleware',

'django.contrib.messages.middleware.MessageMiddleware',

'django.middleware.clickjacking.XFrameOptionsMiddleware',

]

ROOT_URLCONF = 'django_aws_lambda.urls'

TEMPLATES = [

{

'BACKEND': 'django.template.backends.django.DjangoTemplates',

'DIRS': [

str(ROOT_DIR / 'templates'),

str(ROOT_DIR / 'staticfiles'),

],

'OPTIONS': {

'loaders': [

'django.template.loaders.filesystem.Loader',

'django.template.loaders.app_directories.Loader',

],

'context_processors': [

'django.template.context_processors.debug',

'django.template.context_processors.request',

'django.contrib.auth.context_processors.auth',

'django.template.context_processors.i18n',

'django.template.context_processors.media',

'django.template.context_processors.static',

'django.template.context_processors.tz',

'django.contrib.messages.context_processors.messages',

],

},

},

]

WSGI_APPLICATION = 'django_aws_lambda.wsgi.application'

# Database

# https://docs.djangoproject.com/en/3.0/ref/settings/#databases

DATABASES = {

'default': {

'ENGINE': 'django.db.backends.sqlite3',

'NAME': ROOT_DIR / "db.sqlite3",

}

}

# Password validation

# https://docs.djangoproject.com/en/3.0/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [

{

'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',

},

{

'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',

},

{

'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',

},

{

'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',

},

]

# Internationalization

# https://docs.djangoproject.com/en/1.11/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_L10N = True

USE_TZ = True

# Static files (CSS, JavaScript, Images)

# https://docs.djangoproject.com/en/1.11/howto/static-files/

STATIC_ROOT = str(ROOT_DIR / 'staticfiles')

STATIC_URL = '/static/'

STATICFILES_DIRS = [str(ROOT_DIR / 'static')]

STATICFILES_FINDERS = [

'django.contrib.staticfiles.finders.FileSystemFinder',

'django.contrib.staticfiles.finders.AppDirectoriesFinder',

]

MEDIA_ROOT = str(ROOT_DIR / 'media')

MEDIA_URL = '/media/'

ADMIN_URL = env('DJANGO_ADMIN_URL')

```

* create `local.py` and `production.py` files inside `django_aws_lambda` folder on the same level as `settings.py`

* add the following lines to `local.py`:

```python

from .settings import * # noqa

DEBUG = True

```

* add the following lines to `production.py`:

```python

from .settings import * # noqa

DEBUG = False

DATABASES["default"] = {

'ENGINE': 'django.db.backends.postgresql',

'NAME': env("DB_NAME"),

'USER': env("DB_USER"),

'PASSWORD': env("DB_PASSWORD"),

'HOST': env("DB_HOST"),

'PORT': '5432',

}

DATABASES["default"]["ATOMIC_REQUESTS"] = True # noqa F405

DATABASES["default"]["CONN_MAX_AGE"] = env.int("CONN_MAX_AGE", default=60) # noqa F405

SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")

SECURE_SSL_REDIRECT = env.bool("DJANGO_SECURE_SSL_REDIRECT", default=False)

SESSION_COOKIE_SECURE = True

CSRF_COOKIE_SECURE = True

SECURE_HSTS_SECONDS = 60

SECURE_HSTS_INCLUDE_SUBDOMAINS = env.bool("DJANGO_SECURE_HSTS_INCLUDE_SUBDOMAINS", default=True)

SECURE_HSTS_PRELOAD = env.bool("DJANGO_SECURE_HSTS_PRELOAD", default=True)

SECURE_CONTENT_TYPE_NOSNIFF = env.bool("DJANGO_SECURE_CONTENT_TYPE_NOSNIFF", default=True)

INSTALLED_APPS += ["storages"] # noqa F405

AWS_KEY = env("AWS_KEY_ID")

AWS_SECRET = env("AWS_SECRET")

AWS_STORAGE_BUCKET_NAME = env("AWS_STORAGE_BUCKET_NAME")

AWS_QUERYSTRING_AUTH = False

_AWS_EXPIRY = 60 * 60 * 24 * 7

AWS_S3_OBJECT_PARAMETERS = {"CacheControl": f"max-age={_AWS_EXPIRY}, s-maxage={_AWS_EXPIRY}, must-revalidate"}

AWS_S3_REGION_NAME = env("AWS_S3_REGION_NAME", default=None)

AWS_S3_CUSTOM_DOMAIN = env("DJANGO_AWS_S3_CUSTOM_DOMAIN", default=None)

aws_s3_domain = AWS_S3_CUSTOM_DOMAIN or f"{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com"

STATICFILES_STORAGE = "django_aws_lambda.utils.StaticRootS3Boto3Storage"

COLLECTFAST_STRATEGY = "collectfast.strategies.boto3.Boto3Strategy"

STATIC_URL = f"https://{aws_s3_domain}/static/"

DEFAULT_FILE_STORAGE = "django_aws_lambda.utils.MediaRootS3Boto3Storage"

MEDIA_URL = f"https://{aws_s3_domain}/media/"

MEDIAFILES_LOCATION = "/media"

STATICFILES_LOCATION = "/static"

TEMPLATES[-1]["OPTIONS"]["loaders"] = [ # type: ignore[index] # noqa F405

(

"django.template.loaders.cached.Loader",

[

"django.template.loaders.filesystem.Loader",

"django.template.loaders.app_directories.Loader",

],

)

]

```

* update `wsgi.py` file in `django_aws_lambda` folder with the following lines:

```python

"""

WSGI config for django_aws_lambda project.

It exposes the WSGI callable as a module-level variable named ``application``.

For more information on this file, see

https://docs.djangoproject.com/en/1.11/howto/deployment/wsgi/

"""

"""

WSGI config for django_aws_lambda project.

It exposes the WSGI callable as a module-level variable named ``application``.

For more information on this file, see

https://docs.djangoproject.com/en/3.0/howto/deployment/wsgi/

"""

import os

from django.core.wsgi import get_wsgi_application

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'django_aws_lambda.production')

application = get_wsgi_application()

```

* update `urls.py` file in `django_aws_lambda` folder with the following lines:

```python

"""django_aws_lambda URL Configuration

The `urlpatterns` list routes URLs to views. For more information please see:

https://docs.djangoproject.com/en/3.0/topics/http/urls/

Examples:

Function views

1. Add an import: from my_app import views

2. Add a URL to urlpatterns: path('', views.home, name='home')

Class-based views

1. Add an import: from other_app.views import Home

2. Add a URL to urlpatterns: path('', Home.as_view(), name='home')

Including another URLconf

1. Import the include() function: from django.urls import include, path

2. Add a URL to urlpatterns: path('blog/', include('blog.urls'))

"""

from django.contrib import admin

from django.urls import path

from hello.views import hello

urlpatterns = [

path('admin/', admin.site.urls),

path('', hello),

path('<path:resource>', hello),

]

```

* update `manage.py` with the following lines:

```python

#!/usr/bin/env python

"""Django's command-line utility for administrative tasks."""

import os

import sys

def main():

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'django_aws_lambda.production')

try:

from django.core.management import execute_from_command_line

except ImportError as exc:

raise ImportError(

"Couldn't import Django. Are you sure it's installed and "

"available on your PYTHONPATH environment variable? Did you "

"forget to activate a virtual environment?"

) from exc

execute_from_command_line(sys.argv)

if __name__ == '__main__':

main()

```

* create a folder `utils` inside `django_aws_lambda`

* create `storages.py` file inside `utils` folder with the following lines:

```bash

from storages.backends.s3boto3 import S3Boto3Storage

class StaticRootS3Boto3Storage(S3Boto3Storage):

location = "static"

default_acl = "public-read"

class MediaRootS3Boto3Storage(S3Boto3Storage):

location = "media"

file_overwrite = False

```

## Run Django project locally

* set environment variable with a path to Django local configuration file

```bash

export DJANGO_SETTINGS_MODULE=django_aws_lambda.local

```

* migrate database changes

```bash

python manage.py migrate

```

* create a superuser in the database

```bash

python manage.py createsuperuser

```

> Then provide a username, user email, password, and confirm the password

* collect static files

```bash

python manage.py collectstatic

```

* run server locally

```bash

python manage.py runserver

```

* go to [http://127.0.0.1:8000/](http://127.0.0.1:8000/) and you will see this:

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w167cb786hfj1rm956in.png)

* go to [http://127.0.0.1:8000/Dev.to](http://127.0.0.1:8000/Dev.to) and you will see this:

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/85gr40o4ipjb7yh7qj83.png)

## Create serverless configuration

* initialize npm:

```bash

npm init

```

* install serverless

```bash

npm install -g serverless

```

* install serverless plugins

```bash

npm install -P serverless-dotenv-plugin

npm install -P serverless-prune-plugin

npm install -P serverless-python-requirements

npm install -P serverless-wsgi

```

* create serverless.yaml file with the following configuration:

```yaml

service: django-aws-lambda

plugins:

- serverless-dotenv-plugin

- serverless-prune-plugin

- serverless-python-requirements

- serverless-wsgi

useDotenv: true

custom:

dotenv:

logging: false

pythonRequirements:

dockerizePip: non-linux

zip: true

fileName: requirements.txt

stage: ${env:STAGE}

wsgi:

app: django_aws_lambda.wsgi.application

packRequirements: false

prune:

automatic: true

number: 3

functions:

- app:

handler: wsgi_handler.handler

events:

- http: ANY /

- http: ANY /{proxy+}

timeout: 30

provider:

name: aws

role: arn:aws:iam::<role_id>:role/<role_name>

profile: <your-profile-name> # make sure that you configured aws profile using `aws configure --profile <your-profile-name>`

region: us-east-1

runtime: python3.8

versionFunctions: false

stage: ${env:STAGE}

timeout: 60

vpc:

securityGroupIds:

- <your-security-group-id>

- <your-security-group-id>

subnetIds:

- <your-subnet-id>

- <your-subnet-id>

deploymentBucket:

name: ${env:DEPLOYMENT_BUCKET}

apiGateway:

shouldStartNameWithService: true

lambdaHashingVersion: 20201221

package:

individually:

true

exclude:

- .env

- .git/**

- .github/**

- .serverless/**

- static/**

- .cache/**

- .pytest_cache/**

- node_modules/**

```

## Use Docker for deploying your Django project to AWS Lambda using Serverless

* run Amazon Linux 2 docker image:

```bash

docker run -it -v $(pwd):/root/src/ -v /Users/<your_user>/.aws:/root/.aws amazonlinux:latest bash

```

* install the necessary Unix dependencies:

```bash

yum install sudo -y

sudo yum install -y gcc openssl-devel bzip2-devel libffi-devel wget tar sqlite-devel gcc-c++ make

```

* install node.js version 14:

```bash

curl -sL https://rpm.nodesource.com/setup_14.x | sudo -E bash -

sudo yum install -y nodejs

```

* install Python 3.8.7:

```bash

cd /opt

sudo wget https://www.python.org/ftp/python/3.8.7/Python-3.8.7.tgz

sudo tar xzf Python-3.8.7.tgz

cd Python-3.8.7

sudo ./configure --enable-optimizations

sudo make altinstall

sudo rm -f /opt/Python-3.8.7.tgz

```

* create python and pip aliases:

```bash

alias python='python3.8'

alias pip='pip3.8'

```

* update pip and setuptools:

```bash

pip install --upgrade pip setuptools

```

* install serverless:

```bash

npm install -g serverless

```

* move to project directory

```bash

cd /root/src/

```

* install requirements inside docker container:

```bash

pip install -r requirements.txt

```

* set environment variable with a path to django production configuration file

```bash

export DJANGO_SETTINGS_MODULE=django_aws_lambda.production

```

* migrate database changes

```bash

python manage.py migrate

```

* create a superuser in the database

```bash

python manage.py createsuperuser

```

> Then provide a username, user email, password, and confirm the password

* collect static files to AWS S3 bucket

```bash

python manage.py collectstatic

```

> If you get `NoCredentialsError` from `botocore` you should add to environment variables `AWS_PROFILE`:

```bash

export AWS_PROFILE=<your-aws-profile-name>

```

* install serverless packages from package.json

```bash

npm install

```

* deploy your Django project to AWS Lambda using Serverless

```bash

serverless deploy -s production

```

Your response will look like that:

```

Serverless: Adding Python requirements helper to ....

Serverless: Generated requirements from /root/src/requirements.txt in /root/src/.serverless/requirements.txt...

Serverless: Installing requirements from /root/.cache/serverless-python-requirements/ ...

Serverless: Using download cache directory /root/.cache/serverless-python-requirements/downloadCacheslspyc

Serverless: Running ...

Serverless: Zipping required Python packages for ....

Serverless: Using Python specified in "runtime": python3.8

Serverless: Packaging Python WSGI handler...

Serverless: Packaging service...

Serverless: Excluding development dependencies...

Serverless: Removing Python requirements helper from ....

Serverless: Injecting required Python packages to package...

Serverless: Uploading CloudFormation file to S3...

Serverless: Uploading artifacts...

Serverless: Uploading service app.zip file to S3 (60.48 MB)...

Serverless: Validating template...

Serverless: Updating Stack...

Serverless: Checking Stack update progress...

..........

Serverless: Stack update finished...

Service Information

service: <your-serverless-service-name>

stage: production

region: <your-aws-region>

stack: <your-serverless-service-name>-pronduction

resources: 8

api keys:

None

endpoints:

ANY - https://<some-id>.execute-api.<your-aws-region>.amazonaws.com/production

ANY - https://<some-id>.execute-api.<your-aws-region>.amazonaws.com/production/{proxy+}

functions:

app: <your-serverless-service-name>-production-app

layers:

None

Serverless: Prune: Running post-deployment pruning

Serverless: Prune: Querying for deployed function versions

Serverless: Prune: <your-serverless-service-name>-production-app has 3 additional versions published and 0 aliases, 0 versions selected for deletion

Serverless: Prune: Pruning complete.

Serverless: Removing old service artifacts from S3...

**************************************************************************************************************************************

Serverless: Announcing Metrics, CI/CD, Secrets and more built into Serverless Framework. Run "serverless login" to activate for free..

**************************************************************************************************************************************

```

Now, your Django Project will be available at this URL:

`https://<some-id>.execute-api.<your-aws-region>.amazonaws.com/production`

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y47dnlolnp9vuqtjybyf.png)

Congratulations!

Here is a [link to a GitHub repository](https://github.com/VadymKhodak/django-aws-lambda) with the code shown in this blog post.

If you want to learn more about Django projects on AWS Lambdas follow me on Twitter ([@Vaddimart](https://twitter.com/Vaddimart)) I plan to write a post showing how to create all the necessary AWS resources for this Django project, how to add React.js client to the Django project and more.