django and aws

Storing Django Static and Media Files on Amazon S3

Posted by Michael Herman on Nov 13, 2018

Amazon’s Simple Storage System (S3) provides a simple, cost-effective way to store static files. This tutorial shows how to configure Django to load and serve up static and media files, public and private, via an Amazon S3 bucket.

Main dependencies:

  1. Django v2.1
  2. Docker v18.06.1
  3. Python v3.7

Contents

S3 Bucket

Before beginning, you will need an AWS account. If you’re new to AWS, Amazon provides a free tier with 5GB of S3 storage.

To create an S3 bucket, navigate to the S3 page and click “Create bucket”:

aws s3

Give the bucket a unique, DNS-compliant name and select a region:

aws s3

Click “Next”. Leave the default settings for the remaining options and click the “Create bucket” button. You should now see your bucket back on the main S3 page:

aws s3

IAM Access

Although you could use the AWS root user, it’s best for security to create an IAM user that only has access to S3 or to a specific S3 bucket. What’s more, by setting up a group, it makes it much easier to assign (and remove) access to the bucket. So, we’ll start by setting up a group with limited permissions and then create a user and assign that user to the group.

IAM Group

Within the AWS Console, navigate to the main IAM page and click “Groups” on the sidebar. Then, click the “Create New Group” button, provide a name for the group and then search for and select the built-in policy “AmazonS3FullAccess”:

aws iam

Click the “Next Step” button and then “Create Group” to finish setting up the group:

aws iam

If you’d like to limit access even more, to the specific bucket we just created, create a new policy with the following permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::your-bucket-name",
                "arn:aws:s3:::your-bucket-name/*"
            ]
        }
    ]
}

Be sure to replace your-bucket-name with the actual name. Then, detach the “AmazonS3FullAccess” policy from the group and attach the new policy.

IAM User

Back on the main IAM page, click “Users” and then “Add user”. Define a user name and select “Programmatic access” under the “Access type”:

aws iam

Click the next button to move on to the “Permissions” step. Select the group we just created:

aws iam

Click next again and then click “Create user” to create the new user. You should now see the user’s access key ID and secret access key:

aws iam

Take note of the keys.

Django Project

Clone down the django-docker-s3 repo, and then check out the v1 tag to the master branch:

$ git clone https://github.com/testdrivenio/django-docker-s3 --branch v1 --single-branch
$ cd django-docker-s3
$ git checkout tags/v1 -b master

From the project root, create the images and spin up the Docker containers:

$ docker-compose up -d --build

Once the build is complete, navigate to http://localhost:1337:

app

You should be able to upload an image, and then view the image at http://localhost:1337/mediafiles/IMAGE_FILE_NAME.

The radio buttons, for public vs. private, do not work. We will be adding this functionality in later in this tutorial. Ignore them for now.

Take a quick look at the project structure before moving on:

├── app
│   ├── Dockerfile
│   ├── Pipfile
│   ├── Pipfile.lock
│   ├── entrypoint.sh
│   ├── hello_django
│   │   ├── __init__.py
│   │   ├── settings.py
│   │   ├── urls.py
│   │   └── wsgi.py
│   ├── manage.py
│   ├── mediafiles
│   ├── static
│   │   └── bulma.min.css
│   ├── staticfiles
│   └── upload
│       ├── __init__.py
│       ├── admin.py
│       ├── apps.py
│       ├── migrations
│       │   └── __init__.py
│       ├── models.py
│       ├── templates
│       │   └── upload.html
│       ├── tests.py
│       └── views.py
├── docker-compose.yml
└── nginx
    ├── Dockerfile
    └── nginx.conf

Want to learn how to build this project? Check out the Dockerizing Django with Postgres, Gunicorn, and Nginx blog post.

Django Storages

Next, install django-storages, to use S3 as the main Django storage backend, and boto3, to interact with the AWS API.

Update the Pipfile:

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[packages]

django= "==2.1"
gunicorn= "==19.9.0"
boto3= "==1.9.38"
django-storages= "==1.7.1"

[dev-packages]



[requires]

python_version = "3.7"

Add storages to the INSTALLED_APPS in settings.py:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'upload',
    'storages',
]

Update the images and spin up the new containers:

$ docker-compose up -d --build

Static Files

Moving along, we need to update the handling of static files in settings.py:

STATIC_URL = '/staticfiles/'
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
STATICFILES_DIRS = (os.path.join(BASE_DIR, 'static'),)


MEDIA_URL = '/mediafiles/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'mediafiles')

Replace those settings with the following:

USE_S3 = os.getenv('USE_S3') == 'TRUE'

if USE_S3:
    # aws settings
    AWS_ACCESS_KEY_ID = os.getenv('AWS_ACCESS_KEY_ID')
    AWS_SECRET_ACCESS_KEY = os.getenv('AWS_SECRET_ACCESS_KEY')
    AWS_STORAGE_BUCKET_NAME = os.getenv('AWS_STORAGE_BUCKET_NAME')
    AWS_DEFAULT_ACL = 'public-read'
    AWS_S3_CUSTOM_DOMAIN = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
    AWS_S3_OBJECT_PARAMETERS = {'CacheControl': 'max-age=86400'}
    # s3 static settings
    AWS_LOCATION = 'static'
    STATIC_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{AWS_LOCATION}/'
    STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
else:
    STATIC_URL = '/staticfiles/'
    STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')

STATICFILES_DIRS = (os.path.join(BASE_DIR, 'static'),)

MEDIA_URL = '/mediafiles/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'mediafiles')

Take note of USE_S3 and STATICFILES_STORAGE:

  1. The USE_S3 environment variable is used to turn the S3 storage on (value is TRUE) and off (value is FALSE). So, you could configure two Docker compose files - one for development with S3 off and the other for production with S3 on.
  2. The STATICFILES_STORAGE setting configures Django to automatically add static files to the S3 bucket when the collectstatic command is run.

Review the official django-storages documentation for more info on the above settings and config.

Add the appropriate environment variables to the web service in the docker-compose.yml file:

web:
  build: ./app
  command: gunicorn hello_django.wsgi:application --bind 0.0.0.0:8000
  volumes:
    - ./app/:/usr/src/app/
    - static_volume:/usr/src/app/staticfiles
    - media_volume:/usr/src/app/mediafiles
  expose:
    - 8000
  environment:
    - SECRET_KEY=please_change_me
    - SQL_ENGINE=django.db.backends.postgresql
    - SQL_DATABASE=postgres
    - SQL_USER=postgres
    - SQL_PASSWORD=postgres
    - SQL_HOST=db
    - SQL_PORT=5432
    - DATABASE=postgres
    - USE_S3=TRUE
    - AWS_ACCESS_KEY_ID=UPDATE_ME
    - AWS_SECRET_ACCESS_KEY=UPDATE_ME
    - AWS_STORAGE_BUCKET_NAME=UPDATE_ME
  depends_on:
    - db

Don’t forget to update AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with the user keys that you just created along with the AWS_STORAGE_BUCKET_NAME.

To test, re-build and run the containers:

$ docker-compose down
$ docker-compose up -d --build

This will automatically collect the static files (via the entrypoint.sh file). It should take much longer than usual since it is uploading them to the S3 bucket.

http://localhost:1337 should still render correctly:

app

View the page source to ensure the CSS stylesheet is pulled in from the S3 bucket:

app

Verify that the static files can be seen on the AWS console within the “static” subfolder of the S3 bucket:

aws s3

Media uploads will still hit the local filesystem since we’ve only configured S3 for static files. We’ll work with media uploads shortly.

Finally, update the value of USE_S3 to FALSE and re-build the images to make sure that Django uses the local filesystem for static files. Once done, change USE_S3 back to TRUE.

Public Media Files

To prevent users from overwriting existing static files, media file uploads should be placed in a different subfolder in the bucket. We’ll handle this by creating custom storage classes for each type of storage.

Add a new file called storage_backends.py to the “app/hello_django” folder:

from storages.backends.s3boto3 import S3Boto3Storage
from django.conf import settings


class StaticStorage(S3Boto3Storage):
    location = 'static'
    default_acl = 'public-read'


class PublicMediaStorage(S3Boto3Storage):
    location = 'media'
    default_acl = 'public-read'
    file_overwrite = False

Make the following changes to settings.py:

USE_S3 = os.getenv('USE_S3') == 'TRUE'

if USE_S3:
    # aws settings
    AWS_ACCESS_KEY_ID = os.getenv('AWS_ACCESS_KEY_ID')
    AWS_SECRET_ACCESS_KEY = os.getenv('AWS_SECRET_ACCESS_KEY')
    AWS_STORAGE_BUCKET_NAME = os.getenv('AWS_STORAGE_BUCKET_NAME')
    AWS_DEFAULT_ACL = None
    AWS_S3_CUSTOM_DOMAIN = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
    AWS_S3_OBJECT_PARAMETERS = {'CacheControl': 'max-age=86400'}
    # s3 static settings
    STATIC_LOCATION = 'static'
    STATIC_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{STATIC_LOCATION}/'
    STATICFILES_STORAGE = 'hello_django.storage_backends.StaticStorage'
    # s3 public media settings
    PUBLIC_MEDIA_LOCATION = 'media'
    MEDIA_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{PUBLIC_MEDIA_LOCATION}/'
    DEFAULT_FILE_STORAGE = 'hello_django.storage_backends.PublicMediaStorage'
else:
    STATIC_URL = '/staticfiles/'
    STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
    MEDIA_URL = '/mediafiles/'
    MEDIA_ROOT = os.path.join(BASE_DIR, 'mediafiles')

STATICFILES_DIRS = (os.path.join(BASE_DIR, 'static'),)

With the DEFAULT_FILE_STORAGE setting now set, all FileFields will upload their content to the S3 bucket. Review the remaining settings before moving on.

Next, let’s make a few changes to the upload app.

app/upload/models.py:

from django.db import models


class Upload(models.Model):
    uploaded_at = models.DateTimeField(auto_now_add=True)
    file = models.FileField()

app/upload/views.py:

from django.shortcuts import render
from django.conf import settings
from django.core.files.storage import FileSystemStorage

from .models import Upload


def image_upload(request):
    if request.method == 'POST':
        image_file = request.FILES['image_file']
        image_type = request.POST['image_type']
        if settings.USE_S3:
            upload = Upload(file=image_file)
            upload.save()
            image_url = upload.file.url
        else:
            fs = FileSystemStorage()
            filename = fs.save(image_file.name, image_file)
            image_url = fs.url(filename)
        return render(request, 'upload.html', {
            'image_url': image_url
        })
    return render(request, 'upload.html')

Create the new migration file and then build the new images:

$ docker-compose exec web python manage.py makemigrations
$ docker-compose down
$ docker-compose up -d --build

Test it out! Upload an image at http://localhost:1337. The image should be uploaded to S3 (to the media subfolder) and the image_url should include the S3 url:

app demo

Private Media Files

Add a new class to the storage_backends.py:

class PrivateMediaStorage(S3Boto3Storage):
    location = 'private'
    default_acl = 'private'
    file_overwrite = False
    custom_domain = False

Add the appropriate settings:

USE_S3 = os.getenv('USE_S3') == 'TRUE'

if USE_S3:
    # aws settings
    AWS_ACCESS_KEY_ID = os.getenv('AWS_ACCESS_KEY_ID')
    AWS_SECRET_ACCESS_KEY = os.getenv('AWS_SECRET_ACCESS_KEY')
    AWS_STORAGE_BUCKET_NAME = os.getenv('AWS_STORAGE_BUCKET_NAME')
    AWS_DEFAULT_ACL = None
    AWS_S3_CUSTOM_DOMAIN = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
    AWS_S3_OBJECT_PARAMETERS = {'CacheControl': 'max-age=86400'}
    # s3 static settings
    STATIC_LOCATION = 'static'
    STATIC_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{STATIC_LOCATION}/'
    STATICFILES_STORAGE = 'hello_django.storage_backends.StaticStorage'
    # s3 public media settings
    PUBLIC_MEDIA_LOCATION = 'media'
    MEDIA_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{PUBLIC_MEDIA_LOCATION}/'
    DEFAULT_FILE_STORAGE = 'hello_django.storage_backends.PublicMediaStorage'
    # s3 private media settings
    PRIVATE_MEDIA_LOCATION = 'private'
    PRIVATE_FILE_STORAGE = 'hello_django.storage_backends.PrivateMediaStorage'
else:
    STATIC_URL = '/staticfiles/'
    STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
    MEDIA_URL = '/mediafiles/'
    MEDIA_ROOT = os.path.join(BASE_DIR, 'mediafiles')

STATICFILES_DIRS = (os.path.join(BASE_DIR, 'static'),)

Create a new model in app/upload/models.py:

from django.db import models

from hello_django.storage_backends import PublicMediaStorage, PrivateMediaStorage


class Upload(models.Model):
    uploaded_at = models.DateTimeField(auto_now_add=True)
    file = models.FileField(storage=PublicMediaStorage())


class UploadPrivate(models.Model):
    uploaded_at = models.DateTimeField(auto_now_add=True)
    file = models.FileField(storage=PrivateMediaStorage())

Then, update the view:

from django.shortcuts import render
from django.conf import settings
from django.core.files.storage import FileSystemStorage


from .models import Upload, UploadPrivate


def image_upload(request):
    if request.method == 'POST':
        image_file = request.FILES['image_file']
        image_type = request.POST['image_type']
        if settings.USE_S3:
            if image_type == 'private':
                upload = UploadPrivate(file=image_file)
            else:
                upload = Upload(file=image_file)
            upload.save()
            image_url = upload.file.url
        else:
            fs = FileSystemStorage()
            filename = fs.save(image_file.name, image_file)
            image_url = fs.url(filename)
        return render(request, 'upload.html', {
            'image_url': image_url
        })
    return render(request, 'upload.html')

Again, create the migration file, re-build the images, and spin up the new containers:

$ docker-compose exec web python manage.py makemigrations
$ docker-compose down
$ docker-compose up -d --build

To test, upload a private image at http://localhost:1337. Like a public image, the image should be uploaded to S3 (to the private subfolder) and the image_url should include the S3 URL along with the following query string parameters:

  1. AWSAccessKeyId
  2. Signature
  3. Expires

Essentially, we created a temporary, signed URL that users can access for a specific period of time. You won’t be able to access it directly, without the parameters.

app demo

Conclusion

This post walked you through how to create a bucket on Amazon S3, configure an IAM user and group, and set up Django to upload and serve static files and media uploads to and from S3.

By using S3, you-

  1. Increase the amount of space you have available for static and media files
  2. Decrease the stress on your own server since it no longer has to serve up the files
  3. Can limit access to specific files
  4. Can take advantage of the CloudFront CDN

Let us know if we missed anything or if you have any other tips and tricks. You can find the final code in the django-docker-s3 repo.


Microservices with Docker, Flask, and React

Get the full course. Learn how to build, test, and deploy microservices powered by Docker, Flask, and React!


Table of Contents


Join our mailing list to be notified about course updates and new tutorials.