Serving Static Files from AWS S3

Currently, our web app is serving static files, such as images and CSS files, directly from our server. This is a simple and straightforward way to serve static files, but it won't scale well as our website grows and gains more users. For example, our hard drive will eventually run out of space as we add more static files such as images to our website. Files uploaded by our users, called media files in Django, are also currently stored on our server's hard drive and run the same risk. In addition, we're adding more load to our server by serving large static files instead of just lightweight HTML and JSON, leading to slow response times and potential downtime. To solve this problem, we can use AWS S3, a scalable object storage service. S3 is a good fit for our use case because it's designed to store large amounts of data cheaply and is highly available and durable.

Let's get started by creating a new S3 bucket:

1. Navigate to the AWS S3 console and select Create bucket.
2. Give your bucket a name and ensure Server-side encryption with S3 managed keys is selected.
3. Select ACLs enabled.
4. Turn off Block all public access for now. This will get fixed later when we add a CDN.
5. Then click Create bucket.
6. Next, we need to configure CORS (cross-origin resource sharing) to restrict only our website to access the bucket. In your newly created bucket, navigate to the Permissions tab, scroll down to the CORS configuration, and select Edit. Add the below CORS configuration to the bucket:

   [
       {
           "AllowedHeaders": [
               "*"
           ],
           "AllowedMethods": [
               "POST",
               "PUT",
               "DELETE",
           ],
           "AllowedOrigins": [
               "https://example.com"
           ],
           "ExposeHeaders": [],
           "MaxAgeSeconds": 3600
       }
   ]

At this point, we could start uploading files to our new bucket by clicking the Upload button. However, we'd rather Django automatically collect and upload our static files for us as we develop our website. To do this, we need our EC2 machine to directly access the S3 bucket on our behalf. Let's create a new role that allows reading and writing to S3 buckets that we can assign to our EC2 machine:

1. Navigate to the AWS IAM console and select Roles.
2. Select Create role.
3. Select AWS Service and EC2, then select Next.
4. On the permissions page, select AmazonS3FullAccess and select Next.
5. On the final page, give your role a name like ec2-role and select Create role.

Now that our new role is created, let's assign it to our EC2 machine:

1. Navigate to the AWS EC2 console and select your EC2 machine.
2. Select the Actions dropdown. Under Security, select Modify IAM role.
3. Select the role we just created and select Save.

Great! Now our EC2 machine can read and write to S3 buckets. Next, we need to configure our Django app to use S3 for static files and user-uploaded files (called media files in Django).

1. Install boto3 (a Python library for interacting with AWS services) and django-storages:

   pip install boto3 django-storages

2. Note that we should want our web app to continue serving static files from our local directories during development and only pull static files from S3 during production. Add the following to your Django project's settings.py file:

   ...
   if DEBUG:
       STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
       STATIC_URL = '/staticfiles/'
       STATICFILES_DIRS = os.path.join(BASE_DIR, 'static')
       MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
       MEDIA_URL = '/media/'
   else:
       AWS_STORAGE_BUCKET_NAME = 'your-bucket-name'
       AWS_QUERYSTRING_EXPIRE = 3600
       AWS_S3_CUSTOM_DOMAIN = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
       AWS_S3_FILE_OVERWRITE = False
       AWS_DEFAULT_ACL = None
       STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
       DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
       STATICFILES_DIRS = os.path.join(BASE_DIR, 'static')
       STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
       STATIC_URL = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com/staticfiles/'
       MEDIA_ROOT = os.path.join(BASE_DIR, 'mediafiles')
       MEDIA_URL = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com/mediafiles/'
   ...

Now anytime we update our CSS, JS, images, etc., we can use Django's built-in script to automatically gather these static files from the /static directory, copy them to the staticfiles/ directory, and upload them in bulk to S3:

python manage.py collectstatic

Note that collectstatic is done for us automatically during development—whenever we run python manage.py runserver, our static files are automatically served directly from our static/ directory for convenience.

Great! Now our static files and user-uploaded files are stored securely in S3. We've dramatically reduced the load on our server and can scale our website to handle more traffic. However, images and other large files may seem slower to load and users who are located thousands of miles from the S3 bucket will experience slow load times. To solve these issues, we should add a CDN (content delivery network) to our website such as AWS Cloudfront, but that's a topic for another blog post.