We use Resque all the time - but on a recent project we came across a slight issue - our jobs were executing so quickly that the overhead of forking on every job being processed was massively effecting the time taken to process the queue.
The first line of thought was to see if we could make the jobs ‘do more’ - but they really were both as concise and as complex as they needed to be!
Luckily, it looks like our problems have already been solved - there are a number of options already in the wild for improving worker performance with fast running jobs.
In no particular order..
You can run workers without forking - as mentioned, for our jobs in this scenario, there is a big performance hit in the forking of the clean state for processing the job. A simple rake task to disable this forking yields significant performance improvements - this rake task does the job. The downside to this approach is that you don’t get a clean state for the workers, ever, as it never forks. You could periodically reap the worker and re-start.
This gem gives each worker the ability to process a number of jobs in each fork - so instead of forking for each job to be processed, it allows a fork to process N jobs before cleaning up and re-forking. The N here can be whatever you choose. We’ve used this gem before, and it does exactly what it sets out to do - and also provides a useful hook to pre and post forking. However, a recent upgrade to a newer version of the Resque gem has caused us issues - I haven’t had time to investigate where the incompatibility may have crept in. There is also the similar, but differently named resque-multi-job-forks from which this gem is descended, which has been updated more recently, but doesn’t include the pre/post forking hooks.
This gem takes an Event Machine based approach to the workers, and there is a great blog post explaining the problems and reasons behind this gem’s creation - we’re just about to give it a go, so hopefully it will be as good as it looks.
There are other options - such as resque-pool, and also others that allow forks to live for a certain amount of time before respawning (I can’t find it now!).