< All Articles

Video thumbnails with FFmpeg and Paperclip

We’ve spent the last few weeks working on a new Rails app for a film and photography site and, as you might expect from my recent posts, we’re using Paperclip to handle file uploads.

Without any more effort than a script/install of Paperclip we get thumbnailing of images and PDFs, but what about thumbnailing the video files that will be uploaded to the site? Enter FFmpeg.

Installing FFmpeg

In case you’re not familiar with FFmpeg, it is an extremely powerful, open source recorder, converter and streamer of audio and video. Using it just to thumbnail videos is an almost criminal underuse of its capabilities.

And installation is a rare treat too: it’s easier on Windows than on Unix! Although there are no official binaries for Windows (and compiling your own would be painful) there are, thankfully, a number of pre-compiled options available. I decided to use this one, which includes everything needed to make the thumbnails.

1. Download and extract the latest build (I extracted the files to C:\Program Files\ffmpeg)
2. Add the folder to your PATH environment variable (this is optional, but if you don’t you’ll need to use the Paperclip :command_path option)

If you’re on Unix then you’re on your own! Although this page should help you get started…

The processor

Here’s the code:

module Paperclip
  class VideoThumbnail < Processor

    attr_accessor :time_offset, :geometry, :whiny

    def initialize(file, options = {}, attachment = nil)
      super
      @time_offset = options[:time_offset] || '-4'
      unless options[:geometry].nil? || (@geometry = Geometry.parse(options[:geometry])).nil?
        @geometry.width = (@geometry.width / 2.0).floor * 2.0
        @geometry.height = (@geometry.height / 2.0).floor * 2.0
        @geometry.modifier = ''
      end
      @whiny = options[:whiny].nil? ? true : options[:whiny]
      @basename = File.basename(file.path, File.extname(file.path))
    end

    def make
      dst = Tempfile.new([@basename, 'jpg'].compact.join("."))
      dst.binmode

      cmd = %Q[-itsoffset #{time_offset} -i "#{File.expand_path(file.path)}" -y -vcodec mjpeg -vframes 1 -an -f rawvideo]
      cmd << "-s #{geometry.to_s} " unless geometry.nil?
      cmd << %Q["#{File.expand_path(dst.path)}"]

      begin
        success = Paperclip.run('ffmpeg', cmd)
      rescue PaperclipCommandLineError
        raise PaperclipError, "There was an error processing the thumbnail for #{@basename}" if whiny
      end
      dst
    end
  end
end

In order for Paperclip to automatically load the processor it should be saved in the RAILS_ROOT/lib/paperclip_processors directory.

A Paperclip processor must conform to the following rules:

• it must be defined within the Paperclip module as a subclass of Processor
• its constructor (if overriding the one provided by the base class) must accept the file to be processed, a hash of style options and the attachment instance that the file is for
• it must define a make instance method that returns a File or Tempfile containing the processed data

The VideoThumbnail processor accepts three options:

:time_offset

The number of seconds into the video to capture as a thumbnail (this should be a negative number and corresponds to the `itsoffset` option for FFmpeg).

:geometry

Accepts a `wxh` geometry string, ideally both the width and height should be even numbers however if they aren’t the processor will adjust them automatically.

:whiny

Determines whether or not thumbnailing errors are to be reported.

The make instance method of the thumbnailer actually does the work. The code itself is very straightforward:

• it creates a temporary file to store the thumbnail
• builds the options string to pass to FFmpeg
• calls FFmpeg, handling any errors
• it returns the temporary file (a JPEG image)

An example FFmpeg options string used by the thumbnailer looks like this:

-itsoffset -4 -i "C:/Temp/0001.avi" -y -vcodec mjpeg -vframes 1 -an -f rawvideo -s 320x240 "C:/Temp/0001.0001.jpg"

-itsoffset

The time offset (in seconds) to take the thumbnail.

-i

Specifies the input file (passed to the processor by Paperclip).

-y

Overwrites the output file if it exists.

-vcodec mjpeg

Specifies the output codec to use (in this case the mjpeg codec).

-vframes 1

The number of frames to output (we only want one frame as the output is a static image).

-an

Disables audio output.

-f rawvideo

Forces the output format to raw video (no encoding is necessary).

-s 320x240

Specifies the size of the thumbnail.

Patching Paperclip

Paperclip provides the :processors option that, unsurprisingly, allows you to specify the processors to run on a file. Today I submitted a patch that allows this option to accept a Proc that has now been applied to the plugin. By accepting a Proc it makes it possible to specify different processors depending on the type of file uploaded.

If you’re running on an older version of Paperclip and cannot upgrade to the latest version, you can monkey-patch it yourself:

module Paperclip
  class Attachment
    private
      def solidify_style_definitions
        @styles.each do |name, args|
          @styles[name][:geometry] = @styles[name][:geometry].call(instance) if @styles[name][:geometry].respond_to?(:call)
          @styles[name][:processors] = @styles[name][:processors].call(instance) if @styles[name][:processors].respond_to?(:call)
        end
      end
  end
end

In the model I’ve defined a video? method that returns true if the uploaded file is a video and then I use this is in the :processors option like this:

:processors => lambda { |a| a.video? ? [:video_thumbnail] : [:thumbnail] }

Handling file extensions

The final piece of the puzzle is telling Paperclip to use the correct file extension when downloading the thumbnails. Using the :extension interpolation provided by Paperclip won’t work because it uses the file extension of the originally uploaded file (if we upload a ‘.avi’ file then Paperclip would try to load thumbnails with a ‘.avi’ extension too).

My solution was to define a custom interpolation called :content_type_extension in an initializer like this:

Paperclip::Attachment.interpolations[:content_type_extension] = proc do |attachment, style_name|
  case
    when ((style = attachment.styles[style_name]) && !style[:format].blank?) then style[:format]
    when attachment.instance.video? && style_name.to_s != 'original' then 'jpg'
  else
    File.extname(attachment.original_filename).gsub(/^\.+/, "")
  end
end

This essentially does the same thing as the standard :extension interpolation but if the current style is not ‘original’ and the file is a video then it returns a ‘.jpg’ extension.

Putting it all together

So with a processor defined, Paperclip accepting Procs for the :processors option and a custom interpolation to handle the file extension, this is what a model would look like that provides video thumbnailing:

class Attachment {  :small => '36x36#',
                    :medium => '72x72#',
                    :large => '115x115#' },
                    :url => '/:class/:id/:style.:content_type_extension',
                    :path => ':rails_root/assets/:id_partition/:style.:content_type_extension',
                    :processors => lambda { |a| a.video? ? [:video_thumbnail] : [:thumbnail] }

  def video?
    [ 'application/x-mp4',
      'video/mpeg',
      'video/quicktime',
      'video/x-la-asf',
      'video/x-ms-asf',
      'video/x-msvideo',
      'video/x-sgi-movie',
      'video/x-flv',
      'flv-application/octet-stream',
      'video/3gpp',
      'video/3gpp2',
      'video/3gpp-tt',
      'video/BMPEG',
      'video/BT656',
      'video/CelB',
      'video/DV',
      'video/H261',
      'video/H263',
      'video/H263-1998',
      'video/H263-2000',
      'video/H264',
      'video/JPEG',
      'video/MJ2',
      'video/MP1S',
      'video/MP2P',
      'video/MP2T',
      'video/mp4',
      'video/MP4V-ES',
      'video/MPV',
      'video/mpeg4',
      'video/mpeg4-generic',
      'video/nv',
      'video/parityfec',
      'video/pointer',
      'video/raw',
      'video/rtx' ].include?(asset.content_type)
  end

end

In this example the video? method contains the content types that are considered a video file. In your application you will likely want to put this in a configuration file of some sort.

So there you have it: video thumbnailing using FFmpeg and Paperclip. As part of the same project we’ve also managed to implement transcoding video uploads to FLV format using FFmpeg in another Paperclip processor… but that’s a story for another time!

FAQ

How can I get the original file name and content type from within the processor?

You should be able to get the original file name and content type from the attachment instance that’s passed to the processor: in my code above I’d do this in the initialize method of the processor using attachment.original_filename and attachment.instance_read(:content_type). The downside of this approach is that if multiple processors are chained together then the content_type of the file passed to the processor might not be the same as that of the original file.

Another option would be to see if @file.respond_to?(:content_type) and if so use @file.content_type to read it, but again this might not work if multiple processors are used.

A final option would be to use something like the RVideo Inspector class on the file which uses FFmpeg to analyse the content of the file and tell you what it contains: this would work even with chained processors but does of course add a bit of processing overhead.

Does generating the thumbnail using Paperclip.run block the web process? Should a background job be used?

Yes it runs the processor in-process, so it’ll block until it is done. We’ve found thumbnailing video with FFmpeg to be as quick as thumbnailing images with ImageMagick. We also have a Paperclip processor that uses FFmpeg to transcode the video to FLV format: this is a much longer process and for that we use a queue to handle it out-of-process.

Is there a way to determine the dimensions of a video to allow proportional resizing to work?

In one of our projects I added a VideoGeometry class (as a subclass of Paperclip’s own Geometry class). To get the dimensions of the video it uses FFmpeg with just the -i input file argument and captures the output via stderr.

It then uses a simple regular expression to look for the video stream information from the captured output and matches the wxh dimensions it contains.

You could also look at RVideo: it has an Inspector class that does a similar thing: I didn’t use it as my needs were simple, but if you want to get even more information about your videos it’s probably the way to go.