Glenn Gillen

We wanted to push some long running tasks off to the background so that we didn't tie up a mongrel needlessly. I've wrestled (and won!) with backgroundrb in the past, but it just seemed like a chore. And they've since changed the API enough to mean it would be back to the drawing board, so we may as well assess some of the other options. So without too much further ado, the complete guide to installing, using, and monitoring starling.

For those that haven't heard Starling is the back-end queuing system that Twitter use. And despite the stability issues Twitter have, if initial impressions are anything to go by I don't think Starling is the reason. It's fairly light, sits on top of MemCache, and is very quick. The initial lure to Starling though was via the Workling plug-in. We could get a prototype out the door using just the Spawn workling, and then when a queue became important switch over.

Installing Workling

We're going to take the same approach I did, and get a prototype background process happening with Spawn first. So let's install Workling and Spawn:

script/plugin install http://svn.playtype.net/plugins/workling/ script/plugin install http://spawn.rubyforge.org/svn/spawn/

Now in your environment (environment.rb? an initializer? you decide) add the following:

Workling::Remote.dispatcher = Workling::Remote::Runners::SpawnRunner.new

Done!

Creating A Workling Background Worker

Now it's time to create a worker, basically to define what it is exactly that should be executed in the background. For the same of illustration, I'm just going to loop and create some records. I create the follow app/workers/example_worker.rb file:

class ExampleWorker  options[:some_name]

    end

  end 



end

Calling Your New Workling Background Worker

Now in whichever controller would normally need to start this task, you can place a call to the workling worker:

ExampleWorker.async_create_new_records(:some_name => "This Person")

Workling will create a new method for us to call, by prefixing out method name with async_ to allow us to call it asynchronously. Pass in a hash of values you want to use into the method, workling will add a key named :uid containing the unique identifier assigned to this task.

And now we're done here too! This will use Spawn to fork the job, take it out of the mongrel process, and let your rails stack continue and not hold up your other users.

Installing Starling

Install Starling by grabbing the gem and the MemCache Client (which strangely has a dependency on ZenTest which I need to investigated):

sudo gem install memcache-client starling

Now the dependencies are installed, we need to start up the required services. You're going to need the Starling daemon running, as well as the workling client. I'd rather not have Starling run as root so first I create a user, and then give it somewhere to create it's pid and spool files (you can probably skip this and run as root on your local dev box):

sudo /usr/sbin/adduser -s /sbin/nologin starling

sudo mkdir -p /var/spool/starling

sudo mkdir -p /var/run/starling

sudo chown starling:starling /var/spool/starling

sudo chown starling:starling /var/run/starling

And now time to start up the services:

sudo -u starling starling -P /var/run/starling/starling.pid -d

script/workling_starling_client start

A note of particular importance here. The workling_starling_client will run in whatever environment RAILS_ENV is set to, and is not passed in via a command line option. It also does some fruity things with loading up your workers at run time, so if you're running development and test on the one box you'll need to stop the workling client and restart it each time you want to change environments (or best yet, set your rake:spec/rake:test task to do it for you).

Switching Over To Starling For Background Processing

Switching between background processing systems is pretty simple, we just update the environment setting we made earlier to:

Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new

And that's it, business as usual!

Monitoring the sucker

It is all fine and dandy having it up and running now, but it'd be nice to keep it that way. That's where god comes in and some tomfoolery with our app.god config. If you've not already done so, I suggest you read my previous article on monitoring rails with god which talks about creating a generic config file in your rails app so you don't need to manage god configs for each instance.

I'm going to change the example app.god in that article to look like:

class Rubypond

  attr_reader :ports, :server_names, :workling_client



  def initialize

    @ports = [5000, 5001]

    @server_names = "www.rubypond.com rubypond.com" 

    @workling_client = true

  end

end



@apps 

And within the god.conf on the server I'll add the following methods:

def configure_workling_client(w, rails_root,  app_name)

  w.uid = "mongrel" 

  w.gid = "webserver" 

  w.name = "#{app_name}-workling" 

  w.group = "#{app_name}" 

  w.interval = 30.seconds

  w.start = "RAILS_ENV=production #{rails_root}/script/workling_starling_client start" 

  w.stop = "#{rails_root}/script/workling_starling_client stop" 

  w.restart = "#{rails_root}/script/workling_starling_client restart" 

  w.pid_file = "#{rails_root}/log/workling.pid" 



  w.behavior(:clean_pid_file)

end



def monitor_app(w)

  start_if_not_running(w)

  restart_if_resource_hog(w)

  monitor_lifecycle(w)

end

Okay, so monitor_app is actually just renaming monitor_rails_app to be more generic. Make sure you change references everywhere though. Elsewhere in the god.conf file we had a block similar to the following:

@apps.each do |app|

  app_name = app.class.name.downcase

  rails_root = "/var/www/apps/#{app_name}/current" 



  app.ports.each do |port|

    God.watch do |w|

      configure_mongrel(w, rails_root, app_name, port)

      monitor_rails_app(w)

    end

  end



  God.watch do |w|

    configure_workling_client(w, rails_root, app_name)

    monitor_app(w)

  end if app.respond_to? :workling_client

end

The last God.watch there is where the magic happens. It will look at the app we've defined in app.god, and if we've provided an attribute called :workling_client then it will automatically monitor a workling client for this app. We're not quite done yet though. We also need Starling to be running, but that's not on a per app basis we just have one for the server. So I'm just going to insert this directly into god.conf:

God.watch do |w|

  w.name = "starling" 

  w.group = "starlings" 

  w.uid = "starling" 

  w.gid = "starling" 

  w.interval = 30.seconds # default

  w.start = "/usr/bin/starling -P /var/run/starling/starling.pid -d" 

  w.start_grace = 20.seconds

  w.restart_grace = 20.seconds

  w.pid_file = "/var/run/starling/starling.pid" 

  w.behavior(:clean_pid_file)



  monitor_app(w)

end

Now restart god to take the new config, deploy you changes to your rails app, and cross your fingers ;)