Properly uploading files to Amazon S3

Here is a little script I wrote and I though ought to be shared. I use it to upload static files like images, css and javascript so that they can be served by Amazon S3 instead of the main application server (like Google App Engine).

It’s written in Python and does interesting things like compressing and minifying what needs to be. It takes 3 arguments and as 2 options:

src_folder
path to the local folder containing the static files to upload
destination_bucket_name
name of the S3 bucket to upload to (e.g. static.example.com)
prefix
a prefix to use for the destination key (kind of a folder on the destination bucket, I use it to specify a release version to defeat browser caching)
x
if set, the script will set a far future expiry for all files, otherwise the S3 default will be used (one day if I remember well)
m
if set, the script will minify css and javascript files

First you will have to install some dependencies, namely boto, jsmin and cssmin. Installation procedure will depend on your OS but on my Mac I do the following:

And here is the script itself:

Thanks to Nico for the expiry trick :)

2 thoughts on “Properly uploading files to Amazon S3

  1. John Plumridge

    Hello Nick, thanks for a very nice example of a most useful script – the sort I’m looking for, with compression for a static site.

    I don’t think it does incremental upload, am I right – and if so, could you advise on that?
    Thanks, John

    Reply
    1. Claude

      Hi John,

      If by incremental upload you mean it will upload only the files that changed since last time it is not a very difficult addition.

      Instead of blindly creating a new key you can check if it already exists using bucket.get_key(…) then compare key.last_modified and the file last modification time using os.stat(…).st_mtime

      HTH
      Claude

      Reply

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.