We build Web & Mobile Applications.

< All Articles

A Simple Faith.. Monitoring by God!

I’ll try and get all my puns out of the way up front.. In the past, we’ve used monit to monitor our applications and servers - while we’ve been aware of God for some time - then it was a case of better the devil we knew as we didn’t allocate any time to converting to God.

Now, my new found religion hasn’t been born of any material failings with monit, more a case of getting round to seeing what God is all about.

There is a good tutorial on the God homepage itself on how to monitor a Rails application that is running on mongrel. I’ll assume that you’ve read it to get God up and running, and just mention a couple of extra steps I followed to get MySQL and nginx monitored.

MySQL

As with monit, God uses the PID files generated by applications to monitor the running process, so check where your PID file lives:

mysqladmin variables | grep pid_file

For me, the file lives in /var/run/mysqld and is called mysqld.pid

The next step is to determine how to start, stop and restart MySQL - you probably know this off the top of your head. For me, I always use the service command, for example, to start:

service mysqld start

Next step is to create a configuration file for God - I put mine in /etc/god and call it mysqld.god

God.watch do |w|
  w.name = "mysqld"
  w.interval = 30.seconds # default
  w.start = "service mysqld start"
  w.stop = "service mysqld stop"
  w.restart = "service mysqld restart"
  w.start_grace = 20.seconds
  w.restart_grace = 20.seconds
  w.pid_file = "/var/run/mysqld/mysqld.pid"

  w.behavior(:clean_pid_file)

  # determine the state on startup
  w.transition(:init, { true => :up, false => :start }) do |on|
    on.condition(:process_running) do |c|
      c.running = true
    end
  end

  # determine when process has finished starting
  w.transition([:start, :restart], :up) do |on|
    on.condition(:process_running) do |c|
      c.running = true
    end
    # failsafe
    on.condition(:tries) do |c|
      c.times = 8
      c.within = 2.minutes
      c.transition = :start
    end
  end

  # start if process is not running
  w.transition(:up, :start) do |on|
    on.condition(:process_exits)
  end

  # lifecycle
  w.lifecycle do |on|
    on.condition(:flapping) do |c|
      c.to_state = [:start, :restart]
      c.times = 5
      c.within = 1.minute
      c.transition = :unmonitored
      c.retry_in = 10.minutes
      c.retry_times = 5
      c.retry_within = 2.hours
    end
  end
end

You can then test the new monitoring file to make sure it works by running God in non-daemonized mode. Type the following (substituting your filename for where you saved the config file):

god -c /etc/god/mysqld.god -D

You will then see the status displayed on screen as a rolling log, and if mysqld is running you should see the status transition to ‘up’ - otherwise you will see it attempt to start MySQL. Open another terminal and have a play around, for example stop MySQL and watch God restart it. You can also check the status of your monitors using the status parameter:

god status

nginx

The process for setting up monitoring for nginx is broadly similar. Find the PID file (for me it’s in /var/run/nginx/nginx.pid) and create a monitoring configuration file (for me I used /etc/god/nginx.god). The monitoring file looks similar to the MySQL monitor, with the addition of a HTTP response monitor.

God.watch do |w|
  w.name = "nginx"
  w.interval = 30.seconds # default
  w.start = "service nginx start"
  w.stop = "service nginx stop"
  w.restart = "service nginx restart"
  w.start_grace = 20.seconds
  w.restart_grace = 20.seconds
  w.pid_file = "/var/run/nginx/nginx.pid"

  w.behavior(:clean_pid_file)

  # determine the state on startup
  w.transition(:init, { true => :up, false => :start }) do |on|
    on.condition(:process_running) do |c|
      c.running = true
    end
  end

  # determine when process has finished starting
  w.transition([:start, :restart], :up) do |on|
    on.condition(:process_running) do |c|
      c.running = true
    end
    # failsafe
    on.condition(:tries) do |c|
      c.times = 8
      c.within = 2.minutes
      c.transition = :start
    end
  end

  # start if process is not running
  w.transition(:up, :start) do |on|
    on.condition(:process_exits)
  end

  w.transition(:up, :restart) do |on|
      on.condition(:http_response_code) do |c|
        c.host = 'localhost'
        c.port = 80
        c.path = '/monitor.html'
        c.code_is_not = 200
        c.timeout = 10.seconds
        c.times = [3, 5]
      end
  end

  # lifecycle
  w.lifecycle do |on|
    on.condition(:flapping) do |c|
      c.to_state = [:start, :restart]
      c.times = 5
      c.within = 1.minute
      c.transition = :unmonitored
      c.retry_in = 10.minutes
      c.retry_times = 5
      c.retry_within = 2.hours
    end
  end
end

The only shortcoming I can currently see is that the http_response_code doesn’t have provision for HTTP authentication, so you would have to allow incoming requests from localhost to bypass authentication.

As before, have a play around with the nginx process and make sure that God does what it’s meant to.

Making God Startup Automatically

The final step for me was to get God added to my sytem startup by creating a file in my init.d directory. I normally add things as services so it keeps them tidy, but the method you need to use may vary depending on your OS flavour.

My /etc/init.d/god file looks like this:

#!/bin/bash

## BEGIN INIT INFO
# Provides: god
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: start and stop god
# Description: monitoring by god.
## END INIT INFO

# source function library
. /etc/rc.d/init.d/functions

RETVAL=0
prog="god"

set -e

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DESC="god daemon"
NAME=god
DAEMON=/usr/local/bin/$NAME
CONFIGFILEDIR=/etc/god
PIDFILE=/var/run/god/$NAME.pid
LOGFILE=/var/log/god.log
SCRIPTNAME=/etc/init.d/$NAME

# Gracefully exit if the package has been removed.
test -x $DAEMON || exit 0

d_start() {
  $DAEMON -l $LOGFILE -P $PIDFILE || echo -en "\n already running"
  for file in `ls -1 $CONFIGFILEDIR/*.god`; do $DAEMON load $file; done
}

d_stop() {
  kill -QUIT `cat $PIDFILE` || echo -en "\n not running"
}

d_reload() {
  kill -HUP `cat $PIDFILE` || echo -en "\n can't reload"
}

case "$1" in
  start)
    echo -n "Starting $DESC: $NAME"
    d_start
        echo "."
  ;;
  stop)
    echo -n "Stopping $DESC: $NAME"
    d_stop
        echo "."
  ;;
  reload)
    echo -n "Reloading $DESC configuration..."
    d_reload
        echo "."
  ;;
  restart)
    echo -n "Restarting $DESC: $NAME"
    d_stop
    sleep 5
    d_start
    echo "."
  ;;
  *)
    echo "Usage: $SCRIPTNAME {start|stop|restart|reload}" >&2
    exit 3
  ;;
esac

exit 0

Once you’ve created the file then add it to your services:

chkconfig --add god

And that should be it! God should now start and stop with the system, and you can also use the service command to start, stop and restart God.

A final call to god status shows our worship hasn’t been in vain:

mysqld: up
nginx: up
Updated on 07 February 2019
First published by Chris Anderton on 12 February 2008
© Chris Anderton 2019
"A Simple Faith.. Monitoring by God!" by Chris Anderton at TheWebFellas is licensed under a Creative Commons Attribution 4.0 International License.