In no particular order..
Don't fork!
You can run workers without forking - as mentioned, for our jobs in this scenario, there is a big performance hit in the forking of the clean state for processing the job. A simple rake task to disable this forking yields significant performance improvements - this rake task does the job. The downside to this approach is that you don't get a clean state for the workers, ever, as it never forks. You could periodically reap the worker and re-start.
resque-jobs-per-fork
This gem gives each worker the ability to process a number of jobs in each fork - so instead of forking for each job to be processed, it allows a fork to process N jobs before cleaning up and re-forking. The N here can be whatever you choose. We've used this gem before, and it does exactly what it sets out to do - and also provides a useful hook to pre and post forking. However, a recent upgrade to a newer version of the Resque gem has caused us issues - I haven't had time to investigate where the incompatibility may have crept in. There is also the similar, but differently named resque-multi-job-forks from which this gem is descended, which has been updated more recently, but doesn't include the pre/post forking hooks.
em-resque
This gem takes an Event Machine based approach to the workers, and there is a great blog post explaining the problems and reasons behind this gem's creation - we're just about to give it a go, so hopefully it will be as good as it looks.
Other options
There are other options - such as resque-pool, and also others that allow forks to live for a certain amount of time before respawning (I can't find it now!).


Leave a reply
You can use Markdown in your comment as well as plain HTML. You can use
<filter:jscode lang="ruby">and</filter:jscode>tags to surround code blocks (supported languages are css, html, javascript and ruby). Your email address will not be published.If your comment doesn’t appear immediately after posting it could have been marked as spam. Don’t worry: we regularly check for and approve incorrectly filtered comments so you shouldn’t have to wait too long for it to be shown.