UK Specialists in Ruby on Rails, Exalead, AWS, Consultancy - Blog

Amazon DynamoDB - SSD backed non-relational DB is here!

2012-01-18T12:17:00Z

So - pretty exciting stuff, for us at least - Amazon Dynamo DB (previously mentioned here) - has now seen the light of day, and is appearing within the console. Instructions are linked from within the console.

We're already getting cracking on giving it a good play - we'll report back findings of interest!

We're hiring!

2011-05-03T22:13:00Z

Bucking the trend of interplanetary meltdown, we're looking for some 'awesome' people to join our team. We're fleshing out the full details right now and will be posting ads in all the predictable places in the next few weeks, but in the meantime if you fall into any of the following job descriptions and are interested in applying send your CV and examples of your work through to jobs@thewebfellas.com.

These are all permanent positions, salaries are to be decided based on experience and skills but will be based on market rates, as will the overall package on offer. We're currently based in Fareham, Hampshire but may also be relocating soon - wherever we go will be within 15 miles and will be near a mainline station. So if you're currently London based then you'll need to suffer a nice commute (we've all done it the opposite way round!) or perhaps a nice reloaction to the Hampshire countryside.

Openings:

<dl>

User Interface/Web Designer

We need someone with strong skills in designing web based interfaces and graphics. Our clients require designs that are clean, professional and modern with a sprinkling of innovation and spark. We're also interested in people who dabble in pixel art as it's a current favourite of ours and will give us a head start on some of our current internal projects!

Ruby Developer (x2)

We predominantly work with Ruby based applications. Whether it be Rails, Sinatra or anything else, we need someone proficient in Ruby as a language with a strong foundation in clean, sensible programming. The more skills you have, the more chance you have of getting a job with us - make sure your CV's are full of the things you can do rather than the names of as many technologies you can think of!

</dl>

Experience is beneficial, but if you can demonstrate a keen background in the scene or can bring along a strong portfolio of homegrown projects for review then we're happy to consider you. Qualifications are secondary to skill, attitude and enthusiasm.

We are also happy to accept interns/gap year students falling into any of the above skillsets - send us your details and we'll take a look!

Revisited: Tamper-proof cookies in Rails 3

2011-01-25T18:00:00Z

Here's a revisited post that's fairly short and sweet: way back in 2008 I blogged about my implementation of tamper-proof cookies which used a similar technique to that used by Rails for its cookie-based session store. Back then the solution involved a custom cookie jar, the OpenSSL library to generate a HMAC, overriding the ApplicationController#cookies method and a slightly unorthodox method signature for reading cookie values.

Here’s a recap of the code in action:

# Write a tamper-proof cookie
cookies[:test_protected] = { :value => "protected", :protect_from_forgery => true }

# Read a tamper-proof cookie
cookies[:test_protected, true]
=> "protected"

# Read a raw cookie
# This returns the actual contents of the cookie, including the HMAC, and does not perform tamper checking
cookies[:test_protected]
=> "protected--ecec30eed6e122f3a3d5bb914bdd3cc1da4bd28e"

And here’s the nicer Rails 3 version:

# Write a tamper-proof cookie
cookies.signed[:test_protected] = "protected"

# Read a tamper-proof cookie
cookies.signed[:test_protected]
=> "protected"

# Read the raw cookie (this gives you the encoded value and the HMAC)
cookies[:test_protected]
=> "BAhJIg5wcm90ZWN0ZWQGOgZFRg==--18368dd442679a298bc98267d1d45c3046f636a7"

As you can see the magic happens when you use the signed method to read and write cookie values. There are two differences between my code and the Rails 3 version:

Rails 3 automatically Base64 encodes the cookie value and the HMAC is generated using the encoded rather than the raw value.
When a tampered with cookie is detected Rails 3 will return nil for the value whereas my code raised a TamperedWithCookie exception.

Configuration

The only configuration option you might need to adjust is the secret token used when generating the HMAC. Rails will automatically generate a long, random string when you generate a new app that looks something like this:

# lives in config/initializers/secret_token.rb
MyApp::Application.config.secret_token = 'bdf998dad6dcc939bb3285131f2c902ec6d586ae973c02b05ad5ad2be47ade55054be99717a7c6e4191c3283fe6cedf7555126d9d360b996134f5978bc9520c8'

Normally you’ll not need to change this, but if you do make sure it is sufficiently long (minimum of 30 characters) and random enough so that it can’t be guessed easily, otherwise your tamper-proof cookies become somewhat less useful.

Rails 2.3.6 and newer can do it to

Despite the misleading title of this blog it’s actually not just Rails 3 that has this neat functionality: it first appeared in Rails 2.3.6 last May. Usage is exactly the same, but you will need to set the cookie_verifier_secret setting in an initializer like this:

ActionController::Base.cookie_verifier_secret = 'bdf998dad6dcc939bb3285131f2c902ec6d586ae973c02b05ad5ad2be47ade55054be99717a7c6e4191c3283fe6cedf7555126d9d360b996134f5978bc9520c8'

Remember to set your own secret (don’t just copy this one!), Rails has a handy rake task to help you:

rake secret
=> 54540ac4c22d48e24a88398d360b52465e2457a5d6bee99dc897cf5362098555b0940734a1a58b576c2138eea444be392412e81fb712d961a622b099e0cdae74

Rails 2.3.0 to 2.3.5 with ActiveSupport MessageVerifier

If you’re a fan of the classics you may still be using a version of Rails from 2.3.0 to 2.3.5. Under the hood newer Rails versions are using the ActiveSupport::MessageVerifier class to sign and verify cookies. The good news is that this class first made an appearance in Rails 2.3.0 so you can enjoy tamper-proof cookie goodness too. For example:

class ApplicationController < ActionController::Base
  protected
  
    def read_verified_cookie(name)
      cookie_verifier.verify(cookies[name])
    rescue ActiveSupport::MessageVerifier::InvalidSignature
      nil
    end

    def write_verified_cookie(name, value, options = {})
      cookies[name] = options.merge(:value => cookie_verifier.generate(value))
    end
    
    def cookie_verifier
      # Note: the 'secret' string should really live in a configuration file
      @cookie_verifier ||= ActiveSupport::MessageVerifier.new("54540ac4c22d48e24a88398d360b52465e2457a5d6bee99dc897cf5362098555b0940734a1a58b576c2138eea444be392412e81fb712d961a622b099e0cdae74")
    end
end

If you don’t mind the ActiveSupport dependency then you can also do a similar thing in a Sinatra application too.

And finally if your app is using a really old pre-2.3 version of Rails you might as well stick with my trusty old cookie jar from the original blog, although I’d recommend tweaking the code to remove the TamperedWithCookie exception and to Base64 encode the cookie values to make it easier if you migrate to Rails 3 in the future.

Revisited: roll your own pagination links with will_paginate and Rails 3

2010-08-22T09:04:00Z

With a final release of Rails 3 edging closer every day it seems like a good time to revisit some of my old articles from the last few years and bring them up to date.

Back in the summer of 2008 I wrote about custom link renderers using will_paginate and, as it is still one of the most popular posts on the blog, it’s the one I’ve decided to refresh first. Don’t worry if you haven’t read the original article as I’ll be covering the same things here. So without further ado, let’s get stuck in!

Getting started

October 28th, 2011 - note that will_paginate has moved on again since this blog was written so make sure you read the comments (particularly this one) to save yourself getting angry when things don't work as they should.

Of course before going any further you’ll need to have the Rails 3 release candidate installed. If you haven’t already done this I recommend using rvm to save polluting your existing Rails development environment and as you’d expect Ryan Bates has another of his wonderful RailsCasts to help you set it all up.

You’ll also need the Rails 3 version of will_paginate and as Rails 3 uses bundler to manage gem dependencies you won’t have to install it manually like you used to. Simply add the following to the Gemfile in your application’s root directory and then run the bundle install command to install the gem (along with any others required by your application):

gem "will_paginate", "~> 3.0.pre2"

The standard view helper

As with previous versions the standard view helper is creatively named will_paginate which makes it easy to add pagination links to your views. Here’s an example of its use:

# In the controller
@users = User.paginate(:page => params[:page])

# In the view
<%= will_paginate(@users) %>

This will produce the following HTML markup (I’ve broken it into multiple lines to make it more readable):

<div class="pagination">
  <span class="previous_page disabled">← Previous</span> 
  <em>1</em> 
  <a rel="next" href="/users?page=2">2</a> 
  <a href="/users?page=3">3</a> 
  <a href="/users?page=4">4</a> 
  <a class="next_page" rel="next" href="/users?page=2">Next →</a>
</div>

As you can see the helper uses the fairly common markup of a <div> containing links to page numbers along with previous and next page links. The main difference to previous versions is the use of <em> elements for the current page instead of a <span>, so you’ll need to update your CSS if you’re intending re-using an existing stylesheet.

The helper accepts a hash of options for basic customisation:

<dl>

:container

Determines whether or not the container <div> is included in the generated markup (defaults to true).

:class

Allows the CSS class of the container <div> to be specified (defaults to pagination).

:id

This new option allows an ID attribute to be specified for the container <div>. If set to true then an ID will be auto-generated (similar to the way form_for does) or set to a string to use your own ID.

:previous_label

:next_label

Customises the previous and next page link labels.

:page_links

Determines whether or not the individual page links are included in the generated markup (defaults to true). If false then only previous and next page links are generated.

:inner_window

:outer_window

Customises the number of links displayed at either side of the current page (:inner_window) and at the start and end of the page links (:outer_window). Defaults to 4 and 1 respectively.

:separator

Specifies the text inserted between the individual HTML elements generated by the helper (defaults to a single space).

</dl>

You can also get away without passing a collection to the helper if it follows the naming convention of the controller. So in the above example the @users collection is named after the UsersController, so the helper can be used like this:

<%= will_paginate %>

Doing it your way

The standard view helper is fine, but often you’ll need to do your own thing. That’s when you need to use a custom link renderer, and this is where will_paginate has changed the most for Rails 3.

The WillPaginate::LinkRenderer class has been moved into the ViewHelpers module and split into two separate classes:

LinkRendererBase

The main method of interest in the LinkRendererBase class is windowed_page_numbers: it does the work of figuring out which page numbers should be shown using the :inner_window and :outer_window settings and is a replacement for the windowed_links and visible_page_numbers methods in previous versions of will_paginate. Unlike the methods it replaces, windowed_page_numbers doesn’t actually return any HTML, it returns a simple array of page numbers.

The other method to look at is pagination which calls windowed_page_numbers (if the :page_links option has been set) and then adds :previous_page and :next_page symbols to the start and end of the array. It is then up to the renderer to convert this array into HTML for display.

In most cases it is unlikely that you’ll be directly subclassing the LinkRendererBase class: although I’m sure there’s a good reason for a separate base class, it looks to me like an unnecessary abstraction.

LinkRenderer

The LinkRenderer class is a subclass of LinkRendererBase and as this is where the HTML generation takes place it is what you’ll subclass when creating your custom renderer. It provides the to_html method that is called by the will_paginate view helper and is responsible for converting the array of page links returned by the pagination method (from the base class) into HTML markup.

To work with will_paginate, all your renderer has to do is implement two methods: prepare and to_html, so we’ll take a look at them first.

The prepare method is passed the collection being paginated, a hash of options (from the will_paginate helper), and a reference to the template being rendered. By subclassing the LinkRenderer class the preparation of the renderer is taken care of for you, however if you need to do any additional setup in your custom renderer this is the place to put it, keeping in mind this method is called every time the will_paginate view helper is called.

The to_html method maps the page number array returned by the pagination method (inherited from the LinkRendererBase class) to HTML using the page_number method for the page number links and the previous_page and next_page methods (which in turn call the previous_or_next_page method) for the previous and next page links.

As you will see in the examples below the easiest way to create a custom link renderer is by overriding the pagination method to change the the contents of the array it returns. By default the LinkRenderer class handles values in the array like this:

Integer values are converted into links to that page number, for example [1, 2, 3] would result in links to pages 1, 2 and 3 being generated.
Symbols represent the names of methods in the renderer class that should return HTML. For each symbol in the array the renderer will call the corresponding method (using send) and the HTML that the method returns will be added to the pagination output. For example [ :method_one, :method_two ] will result in the methods method_one and method_two being called. If a method name that matches the symbol doesn’t exist then an exception will be raised.

Also note that there is no restriction on the order of values in the array, so you are free to mix integers and symbols however you see fit.

The actual HTML generation takes place in the link and tag methods which are responsible for generating <a> elements for active and <em> elements for inactive links.

These methods replace the page_link_or_span, page_link and page_span methods from previous versions of will_paginate. Another change from previous versions is that the initialize method is no longer used to set the gap marker (the HTML that is used to fill in the gaps in page numbers), instead you can override the protected gap method in your custom renderer to change the markup:

def gap
  tag(:li, "…", :class => "gap")
end

And now for an example

In addition to the methods described above there are a number of additional methods in the LinkRenderer class that can be overridden to generate the markup you require. To see them in action let’s copy from the original article on will_paginate and say we want to generate markup like this for our pagination:

<ul class="pagination">
  <li class="previous_page"><a href="/users?page=1"> ← Previous</a></li>
  <li><a href="/users?page=1">1</a></li>
  <li class="current">2</li>
  <li><a href="/users?page=3">3</a></li>
  <li class="next_page"><a href="/users?page=3">Next →</a></li>
</ul>

Our custom link renderer will look like this (put the code in the lib folder of your application):

class PaginationListLinkRenderer < WillPaginate::ViewHelpers::LinkRenderer

  protected

    def page_number(page)
      unless page == current_page
        tag(:li, link(page, page, :rel => rel_value(page)))
      else
        tag(:li, page, :class => "current")
      end
    end

    def previous_or_next_page(page, text, classname)
      if page
        tag(:li, link(text, page), :class => classname)
      else
        tag(:li, text, :class => classname + ' disabled')
      end
    end

    def html_container(html)
      tag(:ul, html, container_attributes)
    end

end

The first thing you’ll probably notice is that there was no need to override the to_html method. In previous versions if you wanted to change the element used to contain the pagination links from a <div> to something else you had no choice, but the new html_container method means this is no longer the case. The page_number and previous_or_next_page methods also need to be overridden to wrap the page links in <li> tags. Compare this with the code for previous versions of will_paginate and you can see we have a lot less work to do now:

class PaginationListLinkRenderer < WillPaginate::LinkRenderer

  def to_html
    links = @options[:page_links] ? windowed_links : []

    links.unshift(page_link_or_span(@collection.previous_page, 'previous', @options[:previous_label]))
    links.push(page_link_or_span(@collection.next_page, 'next', @options[:next_label]))

    html = links.join(@options[:separator])
    @options[:container] ? @template.content_tag(:ul, html, html_attributes) : html
  end

protected

  def windowed_links
    visible_page_numbers.map { |n| page_link_or_span(n, (n == current_page ? 'current' : nil)) }
  end

  def page_link_or_span(page, span_class, text = nil)
    text ||= page.to_s
    if page && page != current_page
      page_link(page, text, :class => span_class)
    else
      page_span(page, text, :class => span_class)
    end
  end

  def page_link(page, text, attributes = {})
    @template.content_tag(:li, @template.link_to(text, url_for(page)), attributes)
  end

  def page_span(page, text, attributes = {})
    @template.content_tag(:li, text, attributes)
  end

end

Using the custom renderer

As with previous versions of will_paginate, having created a custom renderer you need to tell the view helper to use it. There are two ways to do this, the first is to pass the :renderer option like this:

<%= will_paginate(@users, :renderer => PaginationListLinkRenderer) %>

The :renderer option accepts a class name string, a class or an instance of a link renderer. If you want your custom link renderer to be the default for all pagination you don’t have to add a :renderer option to all of your will_paginate calls, instead you can specify the default in a Rails initializer like this:

WillPaginate::ViewHelpers.pagination_options[:renderer] = 'PaginationListLinkRenderer'

Your view can then contain <%= will_paginate(@users) %> and your link renderer will be used automatically.

A Rails 3 autoloading gotcha

If you try and use your custom link renderer at this point you’re likely to get an uninitialized constant ActionView::CompiledTemplates::PaginationListLinkRenderer exception. This is because, unlike Rails 2.x, the current release candidate of Rails 3 doesn’t autoload code in the lib directory. There’s a lengthy and at times heated discussion of this over on the Rails bug tracker which has not yet been formally resolved. In the meantime you can restore the old Rails 2.x behaviour by changing the autoload paths in config/application.rb:

config.autoload_paths += %W(#{config.root}/lib)

Restart your server and you should be good to go.

How do I...?

So that covers the basics of creating your own link renderer. If you take a look at the comments on my original article you’ll see some people have asked how to achieve different kinds of pagination markup. To finish I’ll update my replies for use with Rails 3.

Remove page numbers and just have first, previous, next and last page links

Turning off page numbers is a simple case of passing :page_links => false to the will_paginate view helper which means all you get is previous and next page links. But in order to add first and last links as well five things need to be done:

The container_attributes method must be overridden to prevent custom options being output as HTML attributes for the pagination container.
An overridden pagination method that returns an array of symbols and does not call the windowed_page_numbers method.
A first_page method that handles the generation of HTML for the first page link.
A last_page method that handles the generation of HTML for the last page link.
Setting new label options for the first and last page links.

Here’s the code:

class MinimalPaginationLinkRenderer < WillPaginate::ViewHelpers::LinkRenderer

  def container_attributes
    super.except(:first_label, :last_label)
  end

  protected

    def pagination
      [ :first_page, :previous_page, :next_page, :last_page ]
    end

    def first_page
      previous_or_next_page(current_page == 1 ? nil : 1, @options[:first_label], "first_page")
    end

    def last_page
      previous_or_next_page(current_page == total_pages ? nil : total_pages, @options[:last_label], "last_page")
    end

end

As you can see the renderer re-uses the previous_or_next_page method to render the first and last page links as it handles the generation of the correct markup for when the current page is the first or last page. To use this in your view use:

<%= will_paginate(:renderer => MinimalPaginationLinkRenderer, :first_label => "&#8676; First", :last_label => "Last &#8677;") %>

Alternatively you can put these options in an initializer using the WillPaginate::ViewHelpers.pagination_options hash.

Change the order of links

To change the order of links, for example to show the page numbers followed by the previous and next page links, is even more straightforward with the Rails 3 version of will_paginate. All you need to do is override the pagination method to change the order of elements in the array it returns, for example:

class ReorderedPaginationLinkRenderer < WillPaginate::ViewHelpers::LinkRenderer

  protected

    def pagination
      items = @options[:page_links] ? windowed_page_numbers : []
      items.push :previous_page, :next_page
    end

end

The only change here from the method in LinkRendererBase is that the :previous_page symbol is being pushed onto the end of the array instead of being added to the beginning. Similarly if you wanted the previous and next links to appear before the page numbers, just change the second line of the method to items.unshift :previous_page, :next_page. As always, either pass the :renderer option to the view helper or set the global option in an initializer to use this renderer.

Only show previous, current and next page links

This is also a trivial task with will_paginate, here’s the code:

class SinglePagePaginationLinkRenderer < WillPaginate::ViewHelpers::LinkRenderer

  protected

    def pagination
      [ :previous_page, current_page, :next_page ]
    end

end

It’s important to note that the reference to current_page in the array is not a symbol like :previous_page and :next_page but is actually a call to the current_page method provided by the LinkRendererBase class (which delegates to the underlying collection to get the current page number).

This isn’t over yet!

And finally a disclaimer: at the time of writing Rails 3 is only at first release candidate stage and in Rails-land this means things can, and likely will, still change substantially before final release. The will_paginate gem will also continue to be fine tuned as time goes by, so be warned if you’re reading this in 2012 and find that something’s not working, its probably because the APIs have moved on again - and it’ll be time for me to write another new version of this article!

Rails 2.3.8, Rack 1.1 and the curious case of the missing quotes

2010-07-15T17:25:00Z

If you're using Rails 2.3.8 for your application and thought that you were safe after May's comedy of errors produced three point updates in as many days, think again. Unfortunately there's a little bug that can lead to parameters being altered or potentially even truncated without warning.

As Rails is a Rack application it delegates the parsing of request parameters to the Rack::Utils module. This worked fine until for some bizarre reason the Rack team decided that the parse_query and normalize_params methods should “properly” parse quoted string values. I haven't been able to find the discussion that led to this commit, so I don't know what the original intention was.

Although this change was made way back in December last year and was released as part of Rack 1.1 in January, it didn't cause a problem for Rails applications until version 2.3.6 bumped the Rack dependency from 1.0 to 1.1. Applications using other Rack frameworks like Sinatra would likely have been affected sooner as they're able to take advantage of a new Rack release more easily.

We first noticed a problem when deploying a Rails 2.3.5 app with JRuby (which was a real pain to track down as something in JRuby was activating Rack 1.1 even though Rails 2.3.5 uses 1.0) and then it cropped up again in one of our Sinatra apps.

So what's the problem?

It's probably best to show a couple of examples:

Rack::Utils.parse_query("foo=bar")
=> {"foo"=>"bar"} # correct
Rack::Utils.parse_query("foo=\"bar\"")
=> {"foo"=>"bar"} # incorrect! should be {"foo"=>"\"bar\""}

It may not be immediately obvious why this removal of quotes could be a problem. For us, our application was providing search functionality using ThinkingSphinx and we wanted to allow users to search for phrases. Rack's interference meant this wasn't possible:

Rack::Utils.parse_query("search=\"I%20am%20a%20search%20phrase\"")
{"search"=>"I am a search phrase"}

The quotes around the phrase are gone, changing the meaning of the ThinkingSphinx search and causing us headaches.

A ticket has also been created on the Rails tracker that suggests that text submitted in a form can also be prematurely truncated as a result of this bug.

How can it be fixed?

The good news is that the changes that caused this bug were reverted in time for Rack 1.2, the bad news is that at the time of writing Rails is still dependent on the broken 1.1 release. Hopefully this will be changed before Rails 2.3.9 is made available: if it isn't expect 2.3.10 to come out a day or so later!

For existing Rails 2.3.8 applications you won't be able to change the Rack version used by Rails but you will be able to monkey-patch the code. Follow the steps below:

Create an initializer called _run_first.rb in the config/initializers directory of your app (note that the leading underscore is important as Rails loads initializers in alphabetical order, the underscore forces this one to the top of the list)

Add the following code to it:

Dir[File.join(Rails.root, "lib", "patches", "**", "*.rb")].sort.each { |patch| require(patch) }

Create a folder called patches in the lib directory of your Rails application (unless it already exists)
Create a file in this lib/patches directory called rack.rb

Add the following code (or get the gist):

module Rack
  module Utils
    def parse_query(qs, d = nil)
      params = {}

      (qs || '').split(d ? /[#{d}] */n : DEFAULT_SEP).each do |p|
        k, v = p.split('=', 2).map { |x| unescape(x) }
        if cur = params[k]
          if cur.class == Array
            params[k] << v
          else
            params[k] = [cur, v]
          end
        else
          params[k] = v
        end
      end

      return params
    end
    module_function :parse_query

    def normalize_params(params, name, v = nil)
      name =~ %r(\A[\[\]]*([^\[\]]+)\]*)
      k = $1 || ''
      after = $' || ''

      return if k.empty?

      if after == ""
        params[k] = v
      elsif after == "[]"
        params[k] ||= []
        raise TypeError, "expected Array (got #{params[k].class.name}) for param `#{k}'" unless params[k].is_a?(Array)
        params[k] << v
      elsif after =~ %r(^\[\]\[([^\[\]]+)\]$) || after =~ %r(^\[\](.+)$)
        child_key = $1
        params[k] ||= []
        raise TypeError, "expected Array (got #{params[k].class.name}) for param `#{k}'" unless params[k].is_a?(Array)
        if params[k].last.is_a?(Hash) && !params[k].last.key?(child_key)
          normalize_params(params[k].last, child_key, v)
        else
          params[k] << normalize_params({}, child_key, v)
        end
      else
        params[k] ||= {}
        raise TypeError, "expected Hash (got #{params[k].class.name}) for param `#{k}'" unless params[k].is_a?(Hash)
        params[k] = normalize_params(params[k], after, v)
      end

      return params
    end
    module_function :normalize_params
  end
end

Restart your server for the patch to take effect.

There's two things going on here:

You've now got a nice, simple pattern for adding patches to your Rails application. For example if you later need to patch a bug in ActionView then you could simply make a lib/patches/action_view.rb file containing the code for the patch and it will automatically be loaded when you next start your app.
The patch file overwrites the broken parse_query and normalize_params methods using the fixed code from Rack 1.2.

You can easily verify that the patch is working by firing up a Rails console (using script/console) and trying one of the examples given above. You should get the correct results, if you don't then run through the above steps again making sure you've done everything exactly as described.

And the moral of the story is?

Even with an extensive test suite it is possible for bugs to slip through: in this case the Rack tests were changed to ensure they passed even though the behaviour was wrong, and the Rails tests didn't cover it because the code should already have been tested in the Rack tests. I guess the question is who tests the tests?!

Create a bootable EBS AMI from a running instance

2010-04-19T20:43:00Z

A quick set of notes on how to create a bootable EBS snapshot from a running EC2 instance - for example, an instance that has been started from an S3 backed AMI.

We've had to do this a few times over the last few months - for the benefit of others, we've outlined how we currently do it - this is based on a number of articles that were surfacing at the time of our research, but I don't have the links to hand. If there's a better way out there feel free to jump in!

Create an EBS volume (if converting from an S3 backed AMI then create it as 10Gb in size)
Attach the EBS volume to your running instance
Stop running services that may be locking files, etc, such as MySQL

At this point, you're ready to copy the contents of your root drive onto the EBS volume. A couple of additional points here - it makes sense to stop any services possible to make the transfer as clean as possible. One of our readers has suggested an alternative approach to dd in the comments - i've included it here as an option for completeness.

Option 1: dd

Copy the root drive to the EBS volume - dd bs=65536 if=SOURCE of=TARGET. Replace SOURCE with the boot volume, e.g. /dev/sda1 and TARGET with the EBS volume, e.g. /dev/sdj
fsck TARGET
mkdir /migrate
mount TARGET /migrate

Option 2: rsync

I suspect with this route you'd need to exclude any directories used for mounting other volumes, otherwise your rsync will run out of space on the target device.

mkfs.ext3 TARGET
mount TARGET /migrate
rsync -ax --progress / /migrate
rm -rf /migrate/migrate

And then continue..

At this point, you can make any necessary modifications to the fstab to allow your EBS based AMI to boot - the file can be found in /migrate/etc/fstab

umount TARGET

Create a snapshot of the EBS volume

Register the snapshot as an AMI by running ec2-register -n NAME --architecture ARCH --block-device-mapping /dev/sda1=SNAPSHOT:SIZE:false. There are a number of options here - some of which you can set when starting the instance if you prefer. NAME is the name of the AMI, ARCH is the architecture (i386 or x86_64). The block device mapping paramaters set the EBS snapshot to use as the boot device, the second parameter sets the size to 15Gb and final parameter sets the delete-on-termination attribute to false.

Run an instance based on the newly registered AMI - if you've set the EBS volume size to anything greater then 10Gb then expand the filesystem to the size of the device.

More details, as usual, can be found in the command line reference.

Ooh la la: Paperclip et les European S3 buckets

2010-01-31T09:48:00Z

At the end of my last blog about Paperclip I mentioned that you need to do some patching if you want to use European S3 buckets to store your files. The problem was introduced when Paperclip made the move from RightAWS to Marcel Molina’s AWS::S3 gem. Unfortunately despite several forks containing patches to AWS::S3 and a 4 month old bug report nothing has been done to officially fix the problem.

So my fellow Europeans, what are we to do?

At the moment there are a couple of options that you might like to try, the first is the quick (and slightly dirty) fix I put together a while back to get things working when a deadline was looming. The second involves dumping AWS::S3 and moving to a European-friendly S3 library, which is the direction I’d like to see Paperclip take.

Option #1 - the quick and dirty patches

Five months ago I first hit this problem when moving one of our Rails apps over to S3 and at that time there were only a few forks of AWS::S3 that attempted to support European buckets. For reasons that are hazy to me now, but probably simply because it worked, I chose to borrow a patch from Vlad Romascanu. If you want to monkey-patch the AWS::S3 gem to use Vlad’s changes then do the following (if you prefer you could of course grab a copy of Vlad’s fork and build a Gem from it):

Make sure AWS::S3 is in your environment.rb file using config.gem "aws-s3", :lib => "aws/s3"
Create a file called aws.rb in RAILS_ROOT/lib/patches

Copy and paste the following code into it (or get the gist):

module AWS
  module S3
    class Authentication
      class CanonicalString
        def initialize(request, options = {})
          super()
          @request = request
          @headers = {}
          @options = options
          # "For non-authenticated or anonymous requests. A NotImplemented error result code will be returned if
          # an authenticated (signed) request specifies a Host: header other than 's3.amazonaws.com'"
          # (from http://docs.amazonwebservices.com/AmazonS3/2006-03-01/VirtualHosting.html)
          request['Host'] = AWS::S3::Base.connection.subdomain || DEFAULT_HOST
          build
        end

        private
          def build
            self << "#{request.method}\n"
            ensure_date_is_valid

            initialize_headers
            set_expiry!

            headers.sort_by {|k, _| k}.each do |key, value|
              value = value.to_s.strip
              self << (key =~ self.class.amazon_header_prefix ? "#{key}:#{value}" : value)
              self << "\n"
            end
            self << (AWS::S3::Base.connection.subdomain ? "/#{AWS::S3::Base.connection.subdomain}#{path}" : path)
          end
      end
    end

    class Bucket
      class << self
        private
          def path(name, options = {})
            if name.is_a?(Hash)
              options = name
              name    = nil
            end
            bucket_name(name) == connection.subdomain ? "/#{RequestOptions.process(options).to_query_string}" : "/#{bucket_name(name)}#{RequestOptions.process(options).to_query_string}"
          end
      end
    end

    class Connection
      def subdomain
        http.address[/^(.+)\.#{DEFAULT_HOST}$/, 1]
      end
    end

    class S3Object
      class << self
        def path!(bucket, name, options = {}) #:nodoc:
          # We're using the second argument for options
          if bucket.is_a?(Hash)
            options.replace(bucket)
            bucket = nil
          end
          bucket_name(bucket) == connection.subdomain ? "/#{name}" : "/#{bucket_name(bucket)}/#{name}"
        end
      end
    end
  end
end

Require the patch from an initializer. I typically do this using the following code which loads any Ruby file in the RAILS_ROOT/lib/patches directory:
```
Dir[File.join(Rails.root, 'lib', 'patches', '**', '*.rb')].sort.each { |patch| require(patch) }
```

The above code fixes AWS::S3, but Paperclip also needs some love: it needs to pass a :server option when the storage module establishes a connection to S3, the to_file method needs to be patched to use binary mode on Windows (ah that old chestnut again!) and to handle non-existent objects more gracefully, making it more consistent with the file system storage module. To make the changes:

Create a file called paperclip.rb in RAILS_ROOT/lib/patches

Copy and paste the following code into it (or get the gist):

module Paperclip
  module Storage
    module S3
      # Patch s3 storage initialisation to pass server name to aws/s3
      def self.extended(base)
        require 'aws/s3'
        base.instance_eval do
          @s3_credentials = parse_credentials(@options[:s3_credentials])
          @bucket         = @options[:bucket]         || @s3_credentials[:bucket]
          @bucket         = @bucket.call(self) if @bucket.is_a?(Proc)
          @s3_options     = @options[:s3_options]     || {}
          @s3_permissions = @options[:s3_permissions] || :public_read
          @s3_protocol    = @options[:s3_protocol]    || (@s3_permissions == :public_read ? 'http' : 'https')
          @s3_headers     = @options[:s3_headers]     || {}
          @s3_host_alias  = @options[:s3_host_alias]
          @url            = ":s3_path_url" unless @url.to_s.match(/^:s3.*url$/)
          AWS::S3::Base.establish_connection!( @s3_options.merge(
            :access_key_id => @s3_credentials[:access_key_id],
            :secret_access_key => @s3_credentials[:secret_access_key],
            :server => "#{@bucket}.s3.amazonaws.com"
          ))
        end
        Paperclip.interpolates(:s3_alias_url) do |attachment, style|
          "#{attachment.s3_protocol}://#{attachment.s3_host_alias}/#{attachment.path(style).gsub(%r{^/}, "")}"
        end
        Paperclip.interpolates(:s3_path_url) do |attachment, style|
          "#{attachment.s3_protocol}://s3.amazonaws.com/#{attachment.bucket_name}/#{attachment.path(style).gsub(%r{^/}, "")}"
        end
        Paperclip.interpolates(:s3_domain_url) do |attachment, style|
          "#{attachment.s3_protocol}://#{attachment.bucket_name}.s3.amazonaws.com/#{attachment.path(style).gsub(%r{^/}, "")}"
        end
      end

      # Patch to use binmode on Windows
      def to_file(style = default_style)
        return @queued_for_write[style] if @queued_for_write[style]
        begin
          file = Tempfile.new(path(style))
          file.binmode if file.respond_to?(:binmode)
          file.write(AWS::S3::S3Object.value(path(style), bucket_name))
          file.rewind
        rescue AWS::S3::NoSuchKey
          file.close if file.respond_to?(:close)
          file = nil
        end
        file
      end
    end
  end
end

If you’re using the patch loading code from earlier then that’s all you need to do, otherwise make sure you require this patch file from an initializer.

Having patched everything up, all that remains is to change your has_attached_file definitions to configure them to use S3, for example:

has_attached_file :photo,
                  :styles => { :small => '105x', :large => '415x' },
                  :storage => :s3,
                  :s3_credentials => File.join(Rails.root, 'config', 's3.yml'),
                  :url => ':s3_domain_url',
                  :path => ':attachment/:id/:style:extension'

As you can see the storage module is set to S3 and the :s3_credentials option is configured to use the s3.yml file, which should look something like this:

common: &common
  access_key_id: yourkeyhere
  secret_access_key: your/secret+here

development:
  <<: *common
  bucket: app-name-development

test:
  <<: *common
  bucket: app-name-test

production:
  <<: *common
  bucket: app-name-production

And finally the :url option is set to use the :s3_domain_url interpolation: this is important as European buckets can only be accessed using domain URLs (e.g. http://app-name-bucket.s3.amazonaws.com/object_path) or aliased URLs using CNAMEs and the :s3_alias_url interpolation.

So there you have it: support for European buckets with Paperclip. As workarounds go this is fairly straight forward, but the good news is that there is a slightly simpler solution.

Option #2 - use a different S3 gem (and patch!)

Last week I needed to write some code to manage objects on S3 in a non-Paperclip scenario. Having immediately ruled out AWS::S3 I decided to see what, if any, newer gems were available. A quick search of GemCutter revealed the not-so-imaginatively named S3 gem: with full support for European buckets and a nice, straightforward syntax I decided to give it a try.

One of the first things I tend to do when trying a new gem or plugin is to have a poke about in the source code to get a feel for how nice the code is, and while doing this with S3 I came across a custom storage module for Paperclip (and there’s one for attachment_fu too).

The current version of the S3 gem (0.2.4) doesn’t support time expiring URLs for accessing private objects so, until 0.2.5 is released, using the gem is a little more work than it would otherwise be:

Grab a copy of the gem source using git clone git://github.com/qoobaa/s3.git
Change to the source code directory and use Rake to build and install the gem with rake install.
Add the gem to your application’s environment.rb using config.gem "s3"
Create a file called paperclip.rb in RAILS_ROOT/lib/patches
Copy and paste the contents of the module file into the file you created in step 4 (or get the gist which includes the changes from steps 6 and 7 so you can skip on to step 8).

Add the expiring_url method to the module, the code looks like this:

def expiring_url(style_name = default_style, time = 3600)
  bucket.objects.build(path(style_name)).temporary_url(Time.current + time)
end

The to_file method also needs patching to better handle non-existing objects and if you’re running on Windows you’ll also want to patch it to handle binary files:

def to_file style_name = default_style
  return @queued_for_write[style_name] if @queued_for_write[style_name]
  begin
    file = Tempfile.new(path(style_name))
    file.binmode if file.respond_to?(:binmode)
    file.write(bucket.objects.find(path(style_name)).content)
    file.rewind
  rescue ::S3::Error::NoSuchKey
    file.close if file.respond_to?(:close)
    file = nil
  end
  file
end

Require the patch from an initializer (see option #1 for an explanation of this code).

Dir[File.join(Rails.root, 'lib', 'patches', '**', '*.rb')].sort.each { |patch| require(patch) }

Configure your model in the same way as described in option #1 and your good to go!

The lesser of two evils

At the moment neither of the above options are ideal. The S3 gem has the advantage of being more actively maintained than AWS::S3, and some of the steps to get it working with Paperclip will be simplified when the next version of the gem is released. While both options involve patching, the patches for S3 are much more trivial when compared with those for AWS::S3, reducing the risk for breakage when Paperclip is updated.

I’d really like to see S3 become the official gem for Paperclip so that it just works without all this hassle. I’ve made a start on a full patch for Paperclip that uses the new gem: I’ve just got to get my head around all the mocking and stubbing that goes on in the storage module tests and then it’ll be ready for the guys at ThoughtBot to have a look!

Conditional duplicate key updates with MySQL

2010-01-18T23:30:00Z

In one of our larger Rails apps the sheer volume of data we process means we’ve had to rely more and more on direct SQL queries, denormalised tables and summary tables to speed things up. When updating summary tables we typically use ON DUPLICATE KEY UPDATE, a MySQL extension to INSERT statements since version 4.1, that allows a record to either be inserted or updated in one query.

For example, with this table:

CREATE TABLE daily_events (
  created_on DATE NOT NULL,
  last_event_id INT(11) UNSIGNED NOT NULL,
  last_event_created_at DATETIME NOT NULL,
  PRIMARY KEY (created_on)
);

The purpose of this table is to provide a daily summary of events storing the id and created_at timestamp of the most recent event for the day. When a new event is processed the daily summary table should also be updated, an operation which is ideally suited to an INSERT ON DUPLICATE KEY UPDATE query.

Simple conditional updates

Unfortunately there’s a catch: the app can process events out-of-order meaning updates to the summary table won’t necessarily occur in the order that the events are created. This means that the last_event_id and last_event_created_at fields should only be updated if the event is newer than the last one for the day. In SQL terms it should look like this:

INSERT INTO daily_events (created_on, last_event_id, last_event_created_at)
VALUES ('2010-01-19', 23, '2010-01-19 10:23:11')
ON DUPLICATE KEY UPDATE
last_event_id = VALUES(last_event_id),
last_event_created_at = VALUES(last_event_created_at)
WHERE last_event_created_at < VALUES(last_event_created_at);

The bad news is that this won’t work as MySQL doesn’t allow WHERE clauses in the update portion of the query. In a simple case like this the easiest workaround is to use the IF function:

INSERT INTO daily_events (created_on, last_event_id, last_event_created_at)
VALUES ('2010-01-19', 23, '2010-01-19 10:23:11')
ON DUPLICATE KEY UPDATE
last_event_id = IF(last_event_created_at < VALUES(last_event_created_at), VALUES(last_event_id), last_event_id),
last_event_created_at = IF(last_event_created_at < VALUES(last_event_created_at), VALUES(last_event_created_at), last_event_created_at);

This works by checking if the last_event_created_at timestamp of the event being updated is newer than the current timestamp, if it is then the new value is assigned to the field in the update, otherwise the current value is used.

An important thing to keep in mind when using this approach is that the order in which you update your fields is very important. I was wrongly under the impression that the updates took place in one mass-assignment after the entire query had been interpreted by MySQL. But they’re not: the assignments happen in the order they appear in the query. To give you an example, this query won’t produce the expected result:

INSERT INTO daily_events (created_on, last_event_id, last_event_created_at)
VALUES ('2010-01-19', 23, '2010-01-19 10:23:11')
ON DUPLICATE KEY UPDATE
last_event_created_at = IF(last_event_created_at < VALUES(last_event_created_at), VALUES(last_event_created_at), last_event_created_at),
last_event_id = IF(last_event_created_at < VALUES(last_event_created_at), VALUES(last_event_id), last_event_id);

When the update is executed with a more recent event, the last_event_created_at field will be updated, but the last_event_id field won’t. This is because when the second IF is evaluated last_event_created_at has already been updated so that last_event_created_at is equal to VALUES(last_event_created_at). Crazy huh?!

Conditions using multiple fields

Keeping in mind the importance of the order in which fields are updated, conditions that involve multiple fields are a little trickier. For example, a slightly different summary table could look like this:

CREATE TABLE last_session_user_events (
  user_id INT(11) UNSIGNED NOT NULL,
  last_session_id INT(11) UNSIGNED NOT NULL,
  last_session_created_at DATETIME NOT NULL,
  last_event_id INT(11) UNSIGNED NOT NULL,
  last_event_created_at DATETIME NOT NULL,
  PRIMARY KEY (user_id)
);

This summary table tracks the most recent event for the most recent session for a user. Updates to this table should only be made if the session is newer, or if the session is the same and the event is newer. Again events can be processed out of order and it is possible for multiple sessions to be active at the same time. The query to insert/update looks like this:

INSERT INTO last_session_user_events (user_id, last_session_id, last_session_created_at, last_event_id, last_event_created_at)
VALUES (6, 45, '2010-01-18 23:31:44', 453, '2010-01-19 10:23:11')
ON DUPLICATE KEY UPDATE
last_session_id = IF((last_session_created_at < VALUES(last_session_created_at) OR (last_session_id = VALUES(last_session_id) AND last_event_created_at < VALUES(last_event_created_at))), VALUES(last_session_id), last_session_id),
last_session_created_at = IF((last_session_created_at < VALUES(last_session_created_at) OR (last_session_id = VALUES(last_session_id) AND
last_event_created_at < VALUES(last_event_created_at))), VALUES(last_session_created_at), last_session_created_at),
last_event_id = IF((last_session_created_at < VALUES(last_session_created_at) OR (last_session_id = VALUES(last_session_id) AND last_event_created_at < VALUES(last_event_created_at))), VALUES(last_event_id), last_event_id),
last_event_created_at = IF((last_session_created_at < VALUES(last_session_created_at) OR (last_session_id = VALUES(last_session_id) AND last_event_created_at < VALUES(last_event_created_at))), VALUES(last_event_created_at), last_event_created_at);

Aside from the fact this looks a bit crazy (people who obsess about DRY code will be weeping) it also suffers from the order of evaluation problem I described earlier. In certain situations the result of the IF statement will change within a single query as the updates are made, leading to inconsistent data. To make this work it is necessary evaluate the IF condition once, before any updates occur, and to store the result in a user defined variable that can be referenced in the rest of the query. Here’s the updated query:

INSERT INTO last_session_user_events (user_id, last_session_id, last_session_created_at, last_event_id, last_event_created_at)
VALUES (6, 45, '2010-01-18 23:31:44', 453, '2010-01-19 10:23:11')
ON DUPLICATE KEY UPDATE
last_session_id = IF((@update_record := (last_session_created_at < VALUES(last_session_created_at) OR (last_session_id = VALUES(last_session_id) AND last_event_created_at < VALUES(last_event_created_at)))), VALUES(last_session_id), last_session_id),
last_session_created_at = IF(@update_record, VALUES(last_session_created_at), last_session_created_at),
last_event_id = IF(@update_record, VALUES(last_event_id), last_event_id),
last_event_created_at = IF(@update_record, VALUES(last_event_created_at), last_event_created_at);

As you can see, not only does this ensure the IF statement remains consistent throughout the update, it also cleans up the query making it a little easier on the eyes.

This is just one of many MySQLisms we’ve had to deal with over the last 12 months, no doubt we’ll blog about more of them in the future. Our lawyers have asked us to point out that no DBAs were harmed in the making of the post!

InfiniDB, Infobright and MonetDB - Day 3: MonetDB

2009-11-26T20:30:00Z

Day 3 of my database exploration mission brings me to MonetDB. Binary downloads are available for Debian, Fedora, Ubuntu and (strangely!) Windows! If we still had any Windows users left here at HQ then it'd be a rare treat, but instead (as usual) our platform of choice (Centos 5) isn't directly available in binary form. We downloaded the Fedora source RPMs and built our own - in case they're of any use then i've put them up on a Google Code site for others to download.

After installing the RPMs then you're ready to get started - before you can do anything you have to start the merovingian process (you could either setup an init script, or run the binary manually for now). For information, the instructions say:

merovingian is a daemon process that controls a collection of database servers, i.e. mserver5 processes, each looking after a single physical database. Start this program to gain access to your MonetDB database farm. merovingian is designed to be used in a system initialisation script in production environments.

With merovingian running then you're ready to create a database - for this you use monetdb - and then start the database using the same command for example:

> monetdb create twf
successfully created database 'twf'

> monetdb status
     name        state     
twf            stopped              
  
> monetdb start twf
starting database 'twf'... done

> monetdb status
     name        state     
twf            running

You now have a running database and can connect to it using mclient. This is similar to most command line clients where you can perform changes to your database as well as query for data.

The first step to transferring the database was as usual - inspect the schema on our MySQL database and update it to make the correct use of the supported data types. As with the other systems, there's no support for unsigned values, it also wasn't immediately obvious to me what the maximum length of a varchar is.

With the tables created it was time to try and migrate some data. Given MonetDB has been around for quite a while then there seemed to be pretty scarce resources with any detailed instructions - I couldn't, for example, find any simple migration tools or documentation detailing the best path for migration. I guess this could be because MonetDB is more often tackled by people with bigger brains or with more time to figure things out.

I attempted to use the following to dump data from MySQL:

select * from h into outfile '/dbtmp/tmp/h' fields terminated by "|" enclosed by '"';

And then the following to import into my MonetDB table:

copy 1000000 records into h from '/dbtmp/tmp/h' using delimiters '|','\n', '"'  null as '';

This yielded reasonable results - though I did have to do some tidying up in the middle with sed - in the end I gave up as there were some string values causing me problems, so I decided to rest on it and went to bed!

In the morning I came back to find the merovingian process was dead, and the status of the database was showing as crashed. I started up the processes and took a look at the status - it said the health was 67% so i'm not really sure what's going on with it!

Performance

In the time I had available I was only able to get a 1 million row table imported successfully to play with - a shocking performance I know, but MonetDB was being quite fussy and I wasn't pressing the right buttons! I did run a few tests and also ran them against the same dataset in MySQL for comparison, all are run from cold - i.e. MySQL and MonetDB are both restarted before each query. I don't expect these queries to be representative of real world cases, I was just thinking of some nasty queries that I could throw at a single table in order to cause some pain.

Query 1

MySQL takes 250msec:

sql>select count(*) from h;
+---------+
| L1      |
+=========+
| 1000000 |
+---------+
1 tuple
Timer       1.532 msec 1 rows

Query 2

MySQL takes 420msec:

sql>select count(*) from h group by intcolumn;
+-------+
| L2    |
+=======+
+-------+
65 tuples
Timer     142.260 msec 65 rows

Query 3

MySQL takes 44,000msec:

sql>select count(*) from h group by varcharcolumn;
+-------+
| L1    |
+=======+
+-------+
12743 tuples
Timer    1464.389 msec 12743 rows

Query 4

MySQL takes 37,500msec:

sql>select count(*) as total from h group by varcharcolumn order by total;
+-------+
| L1    |
+=======+
+-------+
12743 tuples
Timer    1496.537 msec 12743 rows

Query 5

MySQL takes 373,000msec (not a typo, it's more than 6 minutes):

sql>select count(*) as total from h group by varcharcolumn, anothervarcharcolumn order by total;
+-------+
| L1    |
+=======+
+-------+
69696 tuples
Timer    4170.520 msec 69696 rows

Summary

Obviously this quick trial of each of these is not comprehensive enough to make any solid comparisons of performance - the next step will be for me to go through and come up with a proper test plan in order to be a little more methodical about things. However, it has given me a good grounding in how the 3 systems compare with respect to installing and getting started. I'll be keeping a close eye on InfiniDB - while not stable enough right now, i'm sure they'll keep things rolling and I look forward to taking another look. If I can overcome the import obstacles and also the different 'feel' of MonetDB then the basic query results make a compelling case for taking a further look - there's also more to learn here with respect to architecture, deployment techniques, monitoring, etc. Finally, Infobright - it would make my life easier if we could use it on an insert/update/delete basis - as it is I think we'd have a tough time getting clients to pay the license fee - perhaps if bundled with something like EC2 instances with a smaller incremental cost then it may be more palletable and help to increase adoption (it may be that Infobright have lots of customers with open wallets - in which case please share them!). In terms of immediate ease of use, with some visible performance improvements, Infobright fits the bill - but until i've had a chance to compare MonetDB and Infobright in a bit more detail then i'll reserve my final judgement!

InfiniDB, Infobright and MonetDB - Day 2: Infobright

2009-11-25T11:32:00Z

Day 2 of my tour of column based storage brings me on to Infobright Community Edition (ICE). The first impressive point was that based on my blog post of yesterday then I already had an email from Mark in Community Relations at Infobright offering help and advice - despite me calling him the wrong name (I was having a bad day!) then he was immediately helpful and also offered to get some of his team to look into my queries.

As an aside, John from Calpont was also kind enough to drop by to respond to some of my points - to me this gives me a warm fuzzy feeling that both Infobright and Calpont are taking the community seriously - I guess for these products to gain traction they need to make sure people can get motoring with them to improve adoption.

Infobright

Infobright is "designed to deliver a scalable data warehouse optimized for analytic queries" - which is exactly what we're looking for. The main shortcoming at first glance for us is the lack of DML in the Community Edition (it is available in the Enterprise edition, a free trial is available). Similar to InfiniDB then there is a slightly more restricted range of data types compared to MySQL - but currently more flexible than InfiniDB. For example - Infobright can support longer varchars - a feature that John from Calpont says is on it's way in InfiniDB.

To get started we downloaded the 64bit RPM direct from the Infobright website - at the time of testing this was ICE v3.2.2. Install was straightforward with the RPM installing first go. Infobright installed onto a different port to MySQL by default, so there was no conflict with the existing MySQL installation. I guess it's also worth pointing out that we're just running with the default configurations for these systems to begin with.

On my Centos 5 system I had to manually do chkconfig --add mysql-ib and was then able to use service mysql-ib start to crank up Infobright. After tweaking our table definitions to take account of the supported data types and values then we were able to create our tables. We were also able to dump the data from our MySQL tables and import them using the Infobright loader with relative ease.

Performance

With our tables populated we were ready to get querying. It's only fair to point out that this is based on taking our MySQL tables straight into Infobright with no tweakage - as such it probably won't take full advantage of the features on offer. Our cold run figures are based on an average of 5 hopefully cold runs - i.e. we restart MySQL/Infobright in the hope of clearing the caches.

Query 1

SELECT   soh.co, 
         h.ca, 
         o.i, 
         o.n, 
         so.r, 
         so.a, 
         so.w, 
         soh.lhi, 
         soh.lca, 
         soh.lsfhi, 
         soh.lsfhca, 
         h.c, 
         h.r, 
         h.c, 
         h.sti, 
         h.st 
FROM     soh 
         INNER JOIN so 
           ON (so.si = soh.si 
               AND so.oi = soh.oi) 
         INNER JOIN o 
           ON o.i = soh.oi 
         INNER JOIN h 
           ON soh.lsfhi = h.i 
WHERE    soh.si = 125 
         AND soh.co BETWEEN "2009-01-01" AND "2009-10-10" 
ORDER BY soh.co DESC, 
         soh.lsfhca DESC
LIMIT    0,30;

I've removed the column names to protect the innocent, on a cold run our InnoDB (XtraDB) table responded in 1.84 secs vs 8.94 secs for Infobright. A second (warm) run gave 1.51 secs vs 4.40 secs.

Query 2

SELECT 
  SQL_CALC_FOUND_ROWS soh.co, 
                      soh.lca, 
                      soh.oi, 
                      soh.lhi, 
                      soh.lsfhi, 
                      soh.lsfhca 
FROM     soh 
WHERE    soh.si = 125 
         AND soh.co BETWEEN "2009-01-01" AND "2009-10-10" 
ORDER BY soh.co DESC, 
         soh.lsfhca DESC
LIMIT    0,30;

A cold run clocked MySQL at 1.62 secs vs 0.08 secs for Infobright. It's also worth pointing out that SQL_CALC_FOUND_ROWS is here to make a note - i.e. Infobright doesn't support it although it doesn't choke on it if included. Removing it made no significant difference to the times in either database.

Query 3

SELECT   soh.co, 
         soh.lca, 
         soh.oi, 
         soh.lhi, 
         soh.lca, 
         soh.lsfhi, 
         soh.lsfhca 
FROM     soh 
WHERE    soh.si = 125 
         AND soh.co BETWEEN "2009-01-01" AND "2009-10-10" 
         AND soh.oi NOT IN (68633,4) 
ORDER BY soh.co DESC, 
         soh.lsfhca DESC 
LIMIT    0,30;

Cold runs of MySQL gave 1.93 secs vs 0.53 for Infobright.

Query 4

SELECT t1.oi, 
       t1.co, 
       t1.pv, 
       t1.rv, 
       t1.tpv, 
       t1.tnv, 
       t1.trv 
FROM   soh AS t1 
       INNER JOIN (SELECT   oi, 
                            Max(co) AS mco 
                   FROM     soh 
                   WHERE    si = 125 
                            AND co <= '2009-10-24 08:12:23' 
                   GROUP BY oi) AS t2 
         ON t1.oi = t2.oi 
            AND t1.co = t2.mco 
WHERE  t1.oi IN (1420,13501,85778,49753, 
                                  288095,21883,21623,1198, 
                                  123438,54699,145,50293, 
                                  38747,63918,1997,35299, 
                                  97864,321139,138088,38578, 
                                  105140,5440,73439,132415, 
                                  452,180846,4384,29049, 
                                  206296,113606) 
       AND t1.si = 125;

MySQL finished in 1.39 secs vs 0.57 secs for Infobright

Summary

It was a real bonus to be able to get up and running with minimal effort - queries were pretty much able to run unmodified for testing, although both the schema and queries may benefit from optimisation to get the best out of Infobright. As a drop in replacement for a struggling MySQL DB it also holds promise - most of the queries we ran were at least twice as quick as MySQL - the exception being the first one which was slower, so we'll need to work out why.

These current tests are in no way exhaustive - as a first pass i'm just getting to the point of getting things installed and in the position to play with so I can see what they're like to work with, and also to work out which ones are worth investing more time in. For a more thorough review on a more easily explainable dataset then checkout Baron's post on the MySQL Performance Blog.

The only downside for us right now is the lack of insert/update/delete in the community edition - it means we'd have to move toward a batch data load and re-architecting to support this would of course involve effort - which then brings questions around using Hadoop back into the thought processes.

I'm also considering taking a look at Greenplum as they now offer the single node edition for free. Otherwise, it's off to MonetDB-land today and my RPMs have just finished compiling for Centos 5 so i'm ready to install!

InfiniDB, Infobright and MonetDB - Day 1: InfiniDB

2009-11-24T09:48:00Z

We're taking a whistlestop tour of some of the column based storage systems out there for a project we're working on (where the use case seems to fit better with this form of storage rather than straight MySQL). After reading through the series of articles on the MySQL Performance Blog then we chose to look at InfiniDB, Infobright and MonetDB - with the two that talk MySQL coming first for ease of integration right now. I'm also going to do this as a three parter - so first up is InfiniDB.

InfiniDB

InfiniDB was recently released as Open Source by Calpont. The science bit says:

InfiniDB Community Edition provides a scale-up analytics database engine for your data warehousing, business intelligence and read-intensive application needs. Enabled via MySQL and purpose-built for an analytical workload with column-oriented technology at its core, the multi-threaded capabilities of InfiniDB Community Edition fully encompass query, transactional support and bulk load operations.

Handily for us, there are RHEL 5 RPMS available to download so we were able to suck them down and install them - for this trial I used version 0.9.5.1. Following the instructions (we believe fully!) yielded a system that wouldn't start (and yes, we had disabled the existing MySQL install to ensure no port conflict!). Wiping everything and reinstalling then presented a working system.

There are some limitations to the data types supported - there is no support for unsigned numbers, and varchars are limited to 255. I expect this is a design decision as in a database structured around using InfiniDB then the strings would be stored in a MyISAM table and referenced.

A nice feature is support for DML operations - so you can do insert, update and delete.

After successfully exporting a table from MySQL and importing it into InfiniDB using their colxml and cpimport tools we had a table of data and were able to run some simple queries, select count(*) for example. However, after a teabreak the queries were no longer running and giving an error:

ERROR 122 (HY000): There was an internal error encountered in the Calpont Engine while processing this query. The query was cancelled. You may resubmit it if you like. The error is lost Connection to ExeMgr. If the problem continues, please contact your system administrator.

Not having the time or patience to dig deeper, I blitzed it once more and did another clean install. I then went to create the same table as earlier and boom! I saw the load climb up and then typically my wireless dropped out, when it came back I was unable to connect back in and our monitors were showing load sitting at 10. We left it 10 minutes, then rebooted. After a reboot, creating the same table took 20 seconds.

We then successfully imported 4 tables of data and proceeded to start our list of queries to test - the first one caused load to rise to 10 again and a reboot was required. This one has been raised as a bug!.

I decided to leave it there for the day. The InfiniDB system looks promising on paper, and has been written by people infinitely more intelligent than I am, but right now it's not stable enough for me to want to go any further. It is only in Alpha right now so bugs are to be expected - i'll definately be revisiting it when it's a bit more mature or when i've got time to go through and explore the bugs properly so I can contribute back.

Next up is Infobright and MonetDB - i'm tackling Infobright today so will see how things go!

I need your designer glasses, your blue jeans and your black turtle-neck sweater

2009-09-13T23:10:00Z

Ok so it’s not quite Schwarzenegger but last week I terminated a twenty year relationship with Microsoft and bought a Mac. Now I just need to get hold of the Apple uniform and I’ll officially be part of the club!

After more than a year of pontification on what exactly to buy as a replacement for my ageing Dell Inspiron laptop, I finally settled on a shiny new 13” MacBook Pro and an even shinier 24” Cinema Display. So far I’m pretty chuffed with my choice.

Here are a few observations after my first week as a Mac user:

Apple know how to do packaging: OK so it doesn’t make the product any more useable but it does add to the warm and fuzzy feeling you get when you unwrap the new toys that you’ve just spent a big wedge of your hard-earned on.
The laptop itself looks and feels really expensive and its light weight makes it perfect for carrying to meetings or working on the move. My old Dell felt heavy before but now it feels like it weighs a tonne.
OS X is a much nicer place to work than Vista (which I’ve never liked which is why I stayed with XP on my laptop) although I am surprised at how much more I have to do in the terminal. There are a lot of handy utilities that I got used to having in Windows that either don’t exist or aren’t as fully featured on the Mac.
After spending so many hours of my life staring at a 15.4” screen with a fairly high 1680x1050 resolution, the 24” screen makes everything look huge. The picture quality really is something to behold: super deep blacks and vibrant colour and it’s bright enough to light up a small town!
I have to wonder which joker decided to move the keys around on the keyboard: I keep ending up with @ signs when I want double quotes. And could it have been the same joker who thought the hash key was so rarely used that it didn’t need to even appear on the keyboard? Whoever it was they’ve obviously never done any programming work. It could easily have replaced either the § or ± symbols that do get to share a key between them and I can't recall needing either of them recently. Oh and to get a hash you need ALT+3…easy eh?
I gave MacVim a quick tryout for my source code editing as I’ve read a few blogs recently about how great Vim is. I’ve not spent nearly long enough with it to make a fair assessment, but it’s a bit too alien for me at the moment. I’m so used to the E text editor on Windows that I think I’ll have to stick with TextMate on the Mac for now.
Ruby and Rails are both much quicker: sometimes I was able to go and make a cup of tea in the time it took Webrick to boot on my Dell, although this is partly because the Dell was so full of junk that it frequently ran out of both disk space and memory. It is also nice to be able to install gems that require compilation without all the hassle that this usually involves on Windows. In theory I should now be able to get more work done and drink less tea.
I was initially worried that having to compile a lot of the tools I use, like Ruby, MySQL and ImageMagick, from source would be a daunting task, but thanks to some excellent tutorials by Dan Benjamin on HiveLogic I’ve been able to do it with no real problems at all.

So far I’ve been really pleased with the experience and the transition has been a lot less painful than I imagined. I’m now faced with the daunting task of sorting through several years of accumulated files and then moving everything off the Dell and onto the Mac, although if my wallet can take the strain I might have to splash out on one of those new fangled SSDs first…

Protecting your Paperclip downloads

2009-08-29T09:20:00Z

Way back last November when I first blogged about Paperclip I included a brief mention of hiding files behind a controller rather than simply putting them in the public directory for all to see. Since then I’ve noticed that the question of how to actually do this has come up regularly over on Rails Forum and a couple of weeks ago I had to figure out how to update some of our code to protect assets that we had migrated from local file system to Amazon S3 storage. So I figured it’s probably a worthwhile technique to share.

The model

I’ll use a slightly updated version of the Track model from the Getting clever with validations section of my original Paperclip blog. Here’s the code:

class Track < ActiveRecord::Base

  has_attached_file :mp3,
                    :url => '/:class/:id/:style.:extension',
                    :path => ':rails_root/assets/:class/:id_partition/:style.:extension'

  validates_attachment_presence :mp3
  validates_attachment_content_type :mp3, :content_type => [ 'application/mp3', 'application/x-mp3', 'audio/mpeg', 'audio/mp3' ]
  validates_attachment_size :mp3, :less_than => 20.megabytes

  def downloadable?(user)
    user != :guest
  end
end

This does the following:

Defines a Track model that has a Paperclip attachment called mp3.
Configures the URL used to access the mp3 asset, for example /track/1/original.mp3. Leaving the style attribute in the URL makes it possible for future versions of the code to do things like generating 10 second previews of the track (using a custom Paperclip processor) that could then be accessed using a URL like /track/1/preview.mp3.
Configures the path where Paperclip will store uploaded files (for example RAILS_ROOT/assets/tracks/000/000/001/original.mp3 where RAILS_ROOT is the path to the root directory of the Rails app) – the important thing here is that the files are stored outside of the /public directory.
Defines validations to ensure an mp3 is uploaded, it has a valid content type and it isn’t too big.
Defines a downloadable? method that can be used to implement user access rights to the track. For simplicity it just allows all logged in users to access the track, however you can replace this with whatever logic your app requires.

Routes and controllers

Now if you try to provide a link to download the mp3 in your view with something like:

link_to('Listen', track.mp3.url)

You’ll get a routing error like this when you click it:

Routing Error
No route matches "/tracks/1/original.mp3" with {:method=>:get}

I typically map the Paperclip URL to a download action in a controller using map.connect in my routes.rb file like this:

ActionController::Routing::Routes.draw do |map|
  map.connect 'tracks/:id/:style.:format', :controller => 'tracks', :action => 'download', :conditions => { :method => :get }
end

There’s no need to use a named route as it’s unlikely you’ll need to reference the route by name, and by mapping to a custom download action you can also provide the standard resourceful CRUD actions in the controller if needed. Puritanical REST fans might insist that downloads are mapped to a separate resource and that you create a new download resource using a POST request: this is probably worth considering if you're intending doing more than simply streaming a file to the client (like logging statistics, updating billing information) otherwise why complicate things?

The TracksController then needs a download action:

class TracksController < ApplicationController

  SEND_FILE_METHOD = :default

  def download
    head(:not_found) and return if (track = Track.find_by_id(params[:id])).nil?
    head(:forbidden) and return unless track.downloadable?(current_user)

    path = track.mp3.path(params[:style])
    head(:bad_request) and return unless File.exist?(path) && params[:format].to_s == File.extname(path).gsub(/^\.+/, '')

    send_file_options = { :type => File.mime_type?(path) }

    case SEND_FILE_METHOD
      when :apache then send_file_options[:x_sendfile] = true
      when :nginx then head(:x_accel_redirect => path.gsub(Rails.root, ''), :content_type => send_file_options[:type]) and return
    end

    send_file(path, send_file_options)
  end

end

There’s quite a lot going on in the controller, here’s a breakdown of the download action:

An attempt is made to find the Track model using the :id parameter, returning a 404 Not Found response if the track doesn’t exist.
The current user is passed to the downloadable? method to determine if the user is allowed to download the track.

Assume here that current_user is a method provided by the application’s authentication code and returns either a User object or the :guest symbol if no user is logged in. The actual code you’ll use here and in the downloadable? method will probably need to be adapted to suit your own authentication code.

The controller returns a 403 Forbidden response if the user is not allowed to download the track.
The path to the mp3 file on the server is then generated for the passed :style parameter.
The controller then ensures that the file exists (it wouldn’t if, for example, an invalid :style parameter was used) and that the requested file extension matches the actual extension of the file. It returns a 400 Bad Request if necessary.
A hash of options to be used with the send_file method is initialised with the MIME type of the mp3 file.

I’m using the mimetype-fu plugin here (installed using ruby script/plugin install git://github.com/mattetti/mimetype-fu.git) rather than the content_type attribute of the mp3 attachment to allow for future functionality where different styles may use different file types.
The controller supports standard streaming, using the Lighttpd/Apache X-Sendfile or the Nginx X-Accel-Redirect methods for downloading the file.

For simplicity the SEND_FILE_METHOD constant is used here to configure the method to use, but in a real application it should be stored in a configuration setting of some sort (I recommend Luke Redpath’s SimpleConfig gemplugin for this kind of thing).

Depending on the configured streaming method the controller then either uses send_file (for standard and X-Sendfile streaming) or responds with a header (for X-Accel-Redirect streaming).

We usually use Nginx for our Rails apps so we use the X-Accel-Redirect method for streaming files with something like this in our Nginx configuration:


location /assets/ {
  root /path/to/rails_root;
  internal;
}

where /path/to/rails_root contains the full path to our Rails application root.

The view

With the Track model providing access control logic via the downloadable? method and a download action in the TracksController handling the streaming of the mp3 file, it is now possible to provide download links in views using the link_to helper with the url method provided by Paperclip. For example, an index view could look like this:

<ul>
<% @tracks.each do |track| %>
  <li><%= link_to('Listen', track.mp3.url) %></li>
<% end %>
</ul>

Scaling up to Amazon S3

The code above works well if your mp3s are being stored on the local file system, but if your site starts to grow and you need to scale both in terms of storage space and download capacity you’ll probably want to move to S3.

Paperclip provides an S3 storage module that until recently made use of the RightAWS gem however version 2.3.1 and newer use AWS::S3 instead. Be warned that this change does cause some problems if you want to use S3 storage buckets located in Europe, so you might want to stick with 2.3.0 if this will be a problem for you. I'll cover both versions in the code below.

Changing the storage module

The first thing to change is the has_attached_file definition in the Track model:

has_attached_file :mp3,
                  :url => ':s3_domain_url',
                  :path => 'assets/:class/:id/:style.:extension',
                  :storage => :s3,
                  :s3_credentials => File.join(Rails.root, 'config', 's3.yml'),
                  :s3_permissions => 'authenticated-read',
                  :s3_protocol => 'http'

The most obvious changes here are the new :storage, :s3_credentials, :s3_permissions and :s3_protocol options which specify the storage module, credentials file location, permissions for uploaded files and communications protocol respectively.

The credentials YAML file is used to specify the access key, secret access key and bucket name for your S3 accounts and should look something like this:

common: &common
  access_key_id: your_access_key_id_goes_here
  secret_access_key: your_secret_access_key_goes_here

development:
  <<: *common
  bucket: app-name-development

test:
  <<: *common
  bucket: app-name-test

production:
  <<: *common
  bucket: app-name-production

This configures separate S3 buckets for each environment which is a good idea to prevent accidently mixing up your test files with your production files!

The :s3_permissions option allows you to specify one of four canned access policies, in this case authenticated-read is the one to use so that read access is only available to the object owner or an authenticated user.

Slightly less obvious are the changes to the :url and :path options: :url is set to ':s3_domain_url' a Paperclip interpolation that uses virtual hosted style bucket access (especially important if using European buckets as this is the preferred method of access); and the :path is used by Paperclip to generate a key name for the object being stored on S3.

With these changes in place new files uploaded to the Track model will be stored in the appropriate S3 bucket using a key generated from the :path option, for example http://app-name-development.s3.amazonaws.com/assets/tracks/1/original.mp3.

No more streaming, time for a redirection

One of the good things about moving to S3 is that your server no longer has to concern itself with the streaming of data to the clients (using techniques like X-SendFile and X-Accel-Redirect help unburden your Rails processes but the server still has to do all the work). Instead of sending file data to the client the download action in the controller now needs to handle permissions checking but then redirect the client to the file on S3 for download.

This does however present a problem as the uploaded files are using the authenticated-read canned access policy which prevents public access to them. Fortunately S3 provides a way to generate an authenticated URL for private content that only works for a specified period of time: our controller can then redirect the client to this temporary URL to initiate the download but if a someone was to obtain the details of the URL and try to access the file at a later date then they’d be out of luck as the URL would have expired.

The updated code looks like this:

def download
  head(:not_found) and return if (track = Track.find_by_id(params[:id])).nil?
  head(:forbidden) and return unless track.downloadable?(current_user)

  path = track.mp3.path(params[:style])
  head(:bad_request) and return unless params[:format].to_s == File.extname(path).gsub(/^\.+/, '')

  redirect_to(AWS::S3::S3Object.url_for(path, track.mp3.bucket_name, :expires_in => 10.seconds))
end

There are two main differences in the controller:

No check is made to ensure that the object exists as this would add the extra overhead of an additional request to S3 – let S3 worry about returning an appropriate response for non-existent objects.
Instead of streaming the file it redirects to a temporary URL generated using the url_for method. The temporary URL is set to expire after 10 seconds which should be long enough for the redirect to initiate the download: it doesn't matter if the download takes longer than 10 seconds as long as it has started it will continue until it’s done.

If you’re using Paperclip 2.3.0 or older with RightAWS the controller redirect should look like this:

redirect_to(track.mp3.s3.interface.get_link(track.mp3.s3_bucket.to_s, path, 10.seconds))

The view revisited

When using local file system storage the url method of the Paperclip attachment could be used to link to the download action of the TracksController but now that the files are hosted on S3 the url method now returns the URL of the S3 bucket, which is not what we want.

There are a couple of choices here, the route could be changed to a named route and then the view could use something like link_to('Listen', download_track_path(track.id, 'original', 'mp3') but I think a nicer approach is to add a new method to the Track model that uses Paperclip interpolations to generate the download URL. And while I’m at it, I can also move the authenticated S3 URL generation out of the controller and into the model too. Here's the complete Track model:

class Track < ActiveRecord::Base

  has_attached_file :mp3,
                    :url => ':s3_domain_url',
                    :path => 'assets/:class/:id/:style.:extension',
                    :storage => :s3,
                    :s3_credentials => File.join(Rails.root, 'config', 's3.yml'),
                    :s3_permissions => 'authenticated-read',
                    :s3_protocol => 'http'

  validates_attachment_presence :mp3
  validates_attachment_content_type :mp3, :content_type => [ 'application/mp3', 'application/x-mp3', 'audio/mpeg', 'audio/mp3' ]
  validates_attachment_size :mp3, :less_than => 20.megabytes

  def downloadable?(user)
    user != :guest
  end

  def download_url(style = nil, include_updated_timestamp = true)
    url = Paperclip::Interpolations.interpolate('/:class/:id/:style.:extension', mp3, style || mp3.default_style)
    include_updated_timestamp && mp3.updated_at ? [url, mp3.updated_at].compact.join(url.include?("?") ? "&" : "?") : url
  end

  def authenticated_url(style = nil, expires_in = 10.seconds)
    AWS::S3::S3Object.url_for(mp3.path(style || mp3.default_style), mp3.bucket_name, :expires_in => expires_in, :use_ssl => mp3.s3_protocol == 'https')
  end

end

Again for RightAWS (Paperclip <= 2.3.0) the authenticated URL generation is slightly different:

def authenticated_url(style = nil, expires_in = 10.seconds)
  mp3.s3.interface.get_link(mp3.s3_bucket.to_s, mp3.path(style || mp3.default_style), expires_in)
end

This then tidies up the download action, the redirect changes to:

redirect_to(track.authenticated_url(params[:style]))

And the view can now link to downloads:

link_to('Listen', track.download_url)

The end

Hopefully I’ve given you enough here for you to start implementing protected downloads in your own Rails apps. Don’t forget that if you’re using European S3 buckets you’ll need to do some patching otherwise you’ll get strange errors like:

AWS::S3::PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

when you try and upload – fixing that properly is a job for another day!

Bugmash!

2009-08-09T14:13:00Z

Well it’s day two of the first ever Rails BugMash and so far I’ve managed to score a sneaky 1,000 points just by updating my one-line binary fixtures test patch. Meanwhile Matt Duncan and Rizwan Reza are on fire with 4,350 and 4,000 points at the time of writing.

Unfortunately the event has coincided with what may be the only nice weekend of the Great British summer, so I’ve been torn between the chance to mash bugs or to enjoy the sunshine. I’m currently trying to combine the two sat out in the garden squinting to see my laptop screen in the glare of the sun!

My next attempt to score some points is an updated patch, now improved and including a test case, for a lack of quoting of aliased table names in SQL joins which has been (too eagerly) marked as resolved even though it’s still broken. If you get the opportunity please do take a look and comment on the ticket as it’d be nice to get it fixed.

After that, I’m hoping to try and sneak my patch for anonymous extension modules for belongs_to and has_one associations into the bugmash as it has been sat on Lighthouse since March and already has three +1s. Even if it isn’t eligible for the bugmash, I still think it’s a worthy patch so again please take a look and comment on the ticket if you get chance.

And of course there are still plenty more tickets tagged with bugmash to be looked at so even if you’ve never contributed to Rails before, now is a pretty good time to start!

Thin, Rails 1.2.3/1.2.6 and ActionController::Dispatcher (NameError)

2009-07-15T22:54:00Z

Please note: This patch has now been applied to the Thin master repository so will be fixed in all future releases.

Whilst trying to get an old Rails app up and running with Thin (Gem version 1.2.2) then I encountered a spot of bother:

load_missing_constant: uninitialized constant ActionController::Dispatcher (NameError)

I can't remember the exact point of change, but remember that Dispatcher changed into ActionController::Dispatcher somewhere along the line - obviously after Rails 1.2.3 and 1.2.6!

There was already a check in place within Thin to see if the call method could be used on the Dispatcher - the trouble being that because ActionController::Dispatcher isn't defined then it fails!

I've submitted a ticket - a simple check to make sure ActionController::Dispatcher is defined - in the meantime then i've also put a patch up as a gist to get things running.