We build Web & Mobile Applications.

< All Articles

Ooh la la: Paperclip et les European S3 buckets

UPDATE! This issue has been resolved in later releases of Paperclip - check the version of Paperclip you are using to make sure you have the latest.

At the end of my last blog about Paperclip I mentioned that you need to do some patching if you want to use European S3 buckets to store your files. The problem was introduced when Paperclip made the move from RightAWS to Marcel Molina’s AWS::S3 gem. Unfortunately despite several forks containing patches to AWS::S3 and a 4 month old bug report nothing has been done to officially fix the problem.

So my fellow Europeans, what are we to do?

At the moment there are a couple of options that you might like to try, the first is the quick (and slightly dirty) fix I put together a while back to get things working when a deadline was looming. The second involves dumping AWS::S3 and moving to a European-friendly S3 library, which is the direction I’d like to see Paperclip take.

Option #1 - the quick and dirty patches

Five months ago I first hit this problem when moving one of our Rails apps over to S3 and at that time there were only a few forks of AWS::S3 that attempted to support European buckets. For reasons that are hazy to me now, but probably simply because it worked, I chose to borrow a patch from Vlad Romascanu. If you want to monkey-patch the AWS::S3 gem to use Vlad’s changes then do the following (if you prefer you could of course grab a copy of Vlad’s fork and build a Gem from it):

  1. Make sure AWS::S3 is in your environment.rb file:

    config.gem "aws-s3", :lib => "aws/s3"
  2. Create a file called aws.rb in RAILS_ROOT/lib/patches

  3. Copy and paste the following code into it (or get the gist):

    module AWS
      module S3
        class Authentication
          class CanonicalString
            def initialize(request, options = {})
              @request = request
              @headers = {}
              @options = options
              # "For non-authenticated or anonymous requests. A NotImplemented error result code will be returned if
              # an authenticated (signed) request specifies a Host: header other than 's3.amazonaws.com'"
              # (from http://docs.amazonwebservices.com/AmazonS3/2006-03-01/VirtualHosting.html)
              request['Host'] = AWS::S3::Base.connection.subdomain || DEFAULT_HOST
              def build
                self <<: "#{request.method}\n"
                headers.sort_by {|k, _| k}.each do |key, value|
                  value = value.to_s.strip
                  self << (key =~ self.class.amazon_header_prefix ? "#{key}:#{value}" : value)
                  self << "\n"
                self << (AWS::S3::Base.connection.subdomain ? "/#{AWS::S3::Base.connection.subdomain}#{path}" : path)
        class Bucket
          class << self
              def path(name, options = {})
                if name.is_a?(Hash)
                  options = name
                  name = nil
                bucket_name(name) == connection.subdomain ? "/#{RequestOptions.process(options).to_query_string}" : "/#{bucket_name(name)}#{RequestOptions.process(options).to_query_string}"
        class Connection
          def subdomain
            http.address[/^(.+)\.#{DEFAULT_HOST}$/, 1]
        class S3Object
          class << self
            def path!(bucket, name, options = {}) #:nodoc:
              # We're using the second argument for options
              if bucket.is_a?(Hash)
                bucket = nil
              bucket_name(bucket) == connection.subdomain ? "/#{name}" : "/#{bucket_name(bucket)}/#{name}"
  4. Require the patch from an initializer. I typically do this using the following code which loads any Ruby file in the RAILS_ROOT/lib/patches directory:

    Dir[File.join(Rails.root, 'lib', 'patches', '**', '*.rb')].sort.each { |patch| require(patch) }

The above code fixes AWS::S3 but Paperclip also needs some love: it needs to pass a :server option when the storage module establishes a connection to S3, the to_file method needs to be patched to use binary mode on Windows (ah that old chestnut again!) and to handle non-existent objects more gracefully, making it more consistent with the file system storage module. To make the changes:

  1. Create a file called paperclip.rb in RAILS_ROOT/lib/patches

  2. Copy and paste the following code into it (or get the gist):

    module Paperclip
      module Storage
        module S3
          # Patch s3 storage initialisation to pass server name to aws/s3
          def self.extended(base)
            require 'aws/s3'
            base.instance_eval do
              @s3_credentials = parse_credentials(@options[:s3_credentials])
              @bucket = @options[:bucket] || @s3_credentials[:bucket]
              @bucket = @bucket.call(self) if @bucket.is_a?(Proc)
              @s3_options = @options[:s3_options] || {}
              @s3_permissions = @options[:s3_permissions] || :public_read
              @s3_protocol = @options[:s3_protocol] || (@s3_permissions == :public_read ? 'http' : 'https')
              @s3_headers = @options[:s3_headers] || {}
              @s3_host_alias = @options[:s3_host_alias]
              @url = ":s3_path_url" unless @url.to_s.match(/^:s3.*url$/)
              AWS::S3::Base.establish_connection!( @s3_options.merge(
                :access_key_id => @s3_credentials[:access_key_id],
                :secret_access_key => @s3_credentials[:secret_access_key],
                :server => "#{@bucket}.s3.amazonaws.com"
            Paperclip.interpolates(:s3_alias_url) do |attachment, style|
              "#{attachment.s3_protocol}://#{attachment.s3_host_alias}/#{attachment.path(style).gsub(%r{^/}, "")}"
            Paperclip.interpolates(:s3_path_url) do |attachment, style|
              "#{attachment.s3_protocol}://s3.amazonaws.com/#{attachment.bucket_name}/#{attachment.path(style).gsub(%r{^/}, "")}"
            Paperclip.interpolates(:s3_domain_url) do |attachment, style|
              "#{attachment.s3_protocol}://#{attachment.bucket_name}.s3.amazonaws.com/#{attachment.path(style).gsub(%r{^/}, "")}"
          # Patch to use binmode on Windows
          def to_file(style = default_style)
            return @queued_for_write[style] if @queued_for_write[style]
              file = Tempfile.new(path(style))
              file.binmode if file.respond_to?(:binmode)
              file.write(AWS::S3::S3Object.value(path(style), bucket_name))
            rescue AWS::S3::NoSuchKey
              file.close if file.respond_to?(:close)
              file = nil
  3. If you’re using the patch loading code from earlier then that’s all you need to do, otherwise make sure you require this patch file from an initializer.

Having patched everything up, all that remains is to change your has_attached_file definitions to configure them to use S3, for example:

has_attached_file :photo,
                  :styles => { :small => '105x', :large => '415x' },
                  :storage => :s3,
                  :s3_credentials => File.join(Rails.root, 'config', 's3.yml'),
                  :url => ':s3_domain_url',
                  :path => ':attachment/:id/:style:extension'

As you can see the storage module is set to S3 and the :s3_credentials option is configured to use the s3.yml file, which should look something like this:

common: &common
  access_key_id: yourkeyhere
  secret_access_key: your/secret+here

  <<: *common
  bucket: app-name-development

  <<: *common
  bucket: app-name-test

  <<: *common
  bucket: app-name-production

And finally the :url option is set to use the :s3_domain_url interpolation: this is important as European buckets can only be accessed using domain URLs (e.g. http://app-name-bucket.s3.amazonaws.com/object_path) or aliased URLs using CNAMEs and the :s3_alias_url interpolation.

So there you have it: support for European buckets with Paperclip. As workarounds go this is fairly straight forward, but the good news is that there is a slightly simpler solution.

Option #2 - use a different S3 gem (and patch!)

Last week I needed to write some code to manage objects on S3 in a non-Paperclip scenario. Having immediately ruled out AWS::S3 I decided to see what, if any, newer gems were available. A quick search of RubyGems revealed the not-so-imaginatively named S3 gem: with full support for European buckets and a nice, straightforward syntax I decided to give it a try.

One of the first things I tend to do when trying a new gem or plugin is to have a poke about in the source code to get a feel for how nice the code is, and while doing this with S3 I came across a custom storage module for Paperclip (and there’s one for attachment_fu too).

The current version of the S3 gem (0.2.4) doesn’t support time expiring URLs for accessing private objects so, until 0.2.5 is released, using the gem is a little more work than it would otherwise be:

  1. Grab a copy of the gem source:

    git clone git://github.com/qoobaa/s3.git
  2. Change to the source code directory and use Rake to build and install the gem with rake install.

  3. Add the gem to your application’s environment.rb:

    config.gem "s3"
  4. Create a file called paperclip.rb in RAILS_ROOT/lib/patches

  5. Copy and paste the contents of the module file into the file you created in step 4 (or get the gist which includes the changes from steps 6 and 7 so you can skip on to step 8).

  6. Add the expiring_url method to the module, the code looks like this:

    def expiring_url(style_name = default_style, time = 3600)
      bucket.objects.build(path(style_name)).temporary_url(Time.current + time)
  7. The to_file method also needs patching to better handle non-existing objects and if you’re running on Windows you’ll also want to patch it to handle binary files:

    def to_file style_name = default_style
      return @queued_for_write[style_name] if @queued_for_write[style_name]
        file = Tempfile.new(path(style_name))
        file.binmode if file.respond_to?(:binmode)
      rescue ::S3::Error::NoSuchKey
        file.close if file.respond_to?(:close)
        file = nil
  8. Require the patch from an initializer (see option #1 for an explanation of this code).

    Dir[File.join(Rails.root, 'lib', 'patches', '**', '*.rb')].sort.each { |patch| require(patch) }
  9. Configure your model in the same way as described in option #1 and your good to go!

The lesser of two evils

At the moment neither of the above options are ideal. The S3 gem has the advantage of being more actively maintained than AWS::S3, and some of the steps to get it working with Paperclip will be simplified when the next version of the gem is released. While both options involve patching, the patches for S3 are much more trivial when compared with those for AWS::S3, reducing the risk for breakage when Paperclip is updated.

I’d really like to see S3 become the official gem for Paperclip so that it just works without all this hassle. I’ve made a start on a full patch for Paperclip that uses the new gem: I’ve just got to get my head around all the mocking and stubbing that goes on in the storage module tests and then it’ll be ready for the guys at ThoughtBot to have a look!

Updated on 07 February 2019
First published by Rob Anderton on 31 January 2010
© Rob Anderton 2019
"Ooh la la: Paperclip et les European S3 buckets" by Rob Anderton at TheWebFellas is licensed under a Creative Commons Attribution 4.0 International License.