We build Web & Mobile Applications.
UPDATE! This issue has been resolved in later releases of Paperclip - check the version of Paperclip you are using to make sure you have the latest.
At the end of my last blog about Paperclip I mentioned that you need to do some patching if you want to use European S3 buckets to store your files. The problem was introduced when Paperclip made the move from RightAWS to Marcel Molina’s AWS::S3 gem. Unfortunately despite several forks containing patches to AWS::S3 and a 4 month old bug report nothing has been done to officially fix the problem.
So my fellow Europeans, what are we to do?
At the moment there are a couple of options that you might like to try, the first is the quick (and slightly dirty) fix I put together a while back to get things working when a deadline was looming. The second involves dumping AWS::S3 and moving to a European-friendly S3 library, which is the direction I’d like to see Paperclip take.
Five months ago I first hit this problem when moving one of our Rails apps over to S3 and at that time there were only a few forks of AWS::S3 that attempted to support European buckets. For reasons that are hazy to me now, but probably simply because it worked, I chose to borrow a patch from Vlad Romascanu. If you want to monkey-patch the AWS::S3 gem to use Vlad’s changes then do the following (if you prefer you could of course grab a copy of Vlad’s fork and build a Gem from it):
Make sure AWS::S3 is in your environment.rb file:
config.gem "aws-s3", :lib => "aws/s3"
Create a file called aws.rb in RAILS_ROOT/lib/patches
Copy and paste the following code into it (or get the gist):
module AWS
module S3
class Authentication
class CanonicalString
def initialize(request, options = {})
super()
@request = request
@headers = {}
@options = options
# "For non-authenticated or anonymous requests. A NotImplemented error result code will be returned if
# an authenticated (signed) request specifies a Host: header other than 's3.amazonaws.com'"
# (from http://docs.amazonwebservices.com/AmazonS3/2006-03-01/VirtualHosting.html)
request['Host'] = AWS::S3::Base.connection.subdomain || DEFAULT_HOST
build
end
private
def build
self <<: "#{request.method}\n"
ensure_date_is_valid
initialize_headers
set_expiry!
headers.sort_by {|k, _| k}.each do |key, value|
value = value.to_s.strip
self << (key =~ self.class.amazon_header_prefix ? "#{key}:#{value}" : value)
self << "\n"
end
self << (AWS::S3::Base.connection.subdomain ? "/#{AWS::S3::Base.connection.subdomain}#{path}" : path)
end
end
end
class Bucket
class << self
private
def path(name, options = {})
if name.is_a?(Hash)
options = name
name = nil
end
bucket_name(name) == connection.subdomain ? "/#{RequestOptions.process(options).to_query_string}" : "/#{bucket_name(name)}#{RequestOptions.process(options).to_query_string}"
end
end
end
class Connection
def subdomain
http.address[/^(.+)\.#{DEFAULT_HOST}$/, 1]
end
end
class S3Object
class << self
def path!(bucket, name, options = {}) #:nodoc:
# We're using the second argument for options
if bucket.is_a?(Hash)
options.replace(bucket)
bucket = nil
end
bucket_name(bucket) == connection.subdomain ? "/#{name}" : "/#{bucket_name(bucket)}/#{name}"
end
end
end
end
end
Require the patch from an initializer. I typically do this using the following code which loads any Ruby file in the RAILS_ROOT/lib/patches directory:
Dir[File.join(Rails.root, 'lib', 'patches', '**', '*.rb')].sort.each { |patch| require(patch) }
The above code fixes AWS::S3 but Paperclip also needs some love: it needs to pass a :server option when the storage module establishes a connection to S3, the to_file method needs to be patched to use binary mode on Windows (ah that old chestnut again!) and to handle non-existent objects more gracefully, making it more consistent with the file system storage module. To make the changes:
Create a file called paperclip.rb in RAILS_ROOT/lib/patches
Copy and paste the following code into it (or get the gist):
module Paperclip
module Storage
module S3
# Patch s3 storage initialisation to pass server name to aws/s3
def self.extended(base)
require 'aws/s3'
base.instance_eval do
@s3_credentials = parse_credentials(@options[:s3_credentials])
@bucket = @options[:bucket] || @s3_credentials[:bucket]
@bucket = @bucket.call(self) if @bucket.is_a?(Proc)
@s3_options = @options[:s3_options] || {}
@s3_permissions = @options[:s3_permissions] || :public_read
@s3_protocol = @options[:s3_protocol] || (@s3_permissions == :public_read ? 'http' : 'https')
@s3_headers = @options[:s3_headers] || {}
@s3_host_alias = @options[:s3_host_alias]
@url = ":s3_path_url" unless @url.to_s.match(/^:s3.*url$/)
AWS::S3::Base.establish_connection!( @s3_options.merge(
:access_key_id => @s3_credentials[:access_key_id],
:secret_access_key => @s3_credentials[:secret_access_key],
:server => "#{@bucket}.s3.amazonaws.com"
))
end
Paperclip.interpolates(:s3_alias_url) do |attachment, style|
"#{attachment.s3_protocol}://#{attachment.s3_host_alias}/#{attachment.path(style).gsub(%r{^/}, "")}"
end
Paperclip.interpolates(:s3_path_url) do |attachment, style|
"#{attachment.s3_protocol}://s3.amazonaws.com/#{attachment.bucket_name}/#{attachment.path(style).gsub(%r{^/}, "")}"
end
Paperclip.interpolates(:s3_domain_url) do |attachment, style|
"#{attachment.s3_protocol}://#{attachment.bucket_name}.s3.amazonaws.com/#{attachment.path(style).gsub(%r{^/}, "")}"
end
end
# Patch to use binmode on Windows
def to_file(style = default_style)
return @queued_for_write[style] if @queued_for_write[style]
begin
file = Tempfile.new(path(style))
file.binmode if file.respond_to?(:binmode)
file.write(AWS::S3::S3Object.value(path(style), bucket_name))
file.rewind
rescue AWS::S3::NoSuchKey
file.close if file.respond_to?(:close)
file = nil
end
file
end
end
end
end
If you’re using the patch loading code from earlier then that’s all you need to do, otherwise make sure you require this patch file from an initializer.
Having patched everything up, all that remains is to change your has_attached_file definitions to configure them to use S3, for example:
has_attached_file :photo,
:styles => { :small => '105x', :large => '415x' },
:storage => :s3,
:s3_credentials => File.join(Rails.root, 'config', 's3.yml'),
:url => ':s3_domain_url',
:path => ':attachment/:id/:style:extension'
As you can see the storage module is set to S3 and the :s3_credentials option is configured to use the s3.yml file, which should look something like this:
common: &common
access_key_id: yourkeyhere
secret_access_key: your/secret+here
development:
<<: *common
bucket: app-name-development
test:
<<: *common
bucket: app-name-test
production:
<<: *common
bucket: app-name-production
And finally the :url option is set to use the :s3_domain_url interpolation: this is important as European buckets can only be accessed using domain URLs (e.g. http://app-name-bucket.s3.amazonaws.com/object_path) or aliased URLs using CNAMEs and the :s3_alias_url interpolation.
So there you have it: support for European buckets with Paperclip. As workarounds go this is fairly straight forward, but the good news is that there is a slightly simpler solution.
Last week I needed to write some code to manage objects on S3 in a non-Paperclip scenario. Having immediately ruled out AWS::S3 I decided to see what, if any, newer gems were available. A quick search of RubyGems revealed the not-so-imaginatively named S3 gem: with full support for European buckets and a nice, straightforward syntax I decided to give it a try.
One of the first things I tend to do when trying a new gem or plugin is to have a poke about in the source code to get a feel for how nice the code is, and while doing this with S3 I came across a custom storage module for Paperclip (and there’s one for attachment_fu too).
The current version of the S3 gem (0.2.4) doesn’t support time expiring URLs for accessing private objects so, until 0.2.5 is released, using the gem is a little more work than it would otherwise be:
Grab a copy of the gem source:
git clone git://github.com/qoobaa/s3.git
Change to the source code directory and use Rake to build and install the gem with rake install.
Add the gem to your application’s environment.rb:
config.gem "s3"
Create a file called paperclip.rb in RAILS_ROOT/lib/patches
Copy and paste the contents of the module file into the file you created in step 4 (or get the gist which includes the changes from steps 6 and 7 so you can skip on to step 8).
Add the expiring_url method to the module, the code looks like this:
def expiring_url(style_name = default_style, time = 3600)
bucket.objects.build(path(style_name)).temporary_url(Time.current + time)
end
The to_file method also needs patching to better handle non-existing objects and if you’re running on Windows you’ll also want to patch it to handle binary files:
def to_file style_name = default_style
return @queued_for_write[style_name] if @queued_for_write[style_name]
begin
file = Tempfile.new(path(style_name))
file.binmode if file.respond_to?(:binmode)
file.write(bucket.objects.find(path(style_name)).content)
file.rewind
rescue ::S3::Error::NoSuchKey
file.close if file.respond_to?(:close)
file = nil
end
file
end
Require the patch from an initializer (see option #1 for an explanation of this code).
Dir[File.join(Rails.root, 'lib', 'patches', '**', '*.rb')].sort.each { |patch| require(patch) }
Configure your model in the same way as described in option #1 and your good to go!
At the moment neither of the above options are ideal. The S3 gem has the advantage of being more actively maintained than AWS::S3, and some of the steps to get it working with Paperclip will be simplified when the next version of the gem is released. While both options involve patching, the patches for S3 are much more trivial when compared with those for AWS::S3, reducing the risk for breakage when Paperclip is updated.
I’d really like to see S3 become the official gem for Paperclip so that it just works without all this hassle. I’ve made a start on a full patch for Paperclip that uses the new gem: I’ve just got to get my head around all the mocking and stubbing that goes on in the storage module tests and then it’ll be ready for the guys at ThoughtBot to have a look!